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Preface 


For  many  years  I  have  been  closely  associated  with 
both  the  scientific  and  the  nanac^erial  aspects  of  research 
and  development  programs*  Within  my  Air  Force  career 
fields  I  have  held  fully  qiialificd  specialty  codes  as  a 
** Developmental  Engineer  for  Aerospace  Systeins<‘  and  as  a 
**Reteereh  and  Development  Officer."  I  was  a  meaiber  of  the 
Wdepont  System  Phasing  Group  C/S-117L)  which  ori^^inally 
planned  many  of  our  present  Air  Force  space  programs. 

Later»  while  assigned  to  the  Office  of  the  Secretary  of 
the  Air  Force»  I  served  as  a  project  officer  on  various 
space  programs  with  the  OSAFU  Special  Projects  Office  and 
subsequently  with  the  Air  Force  Systems  Commandos  Space 
Systems  Division*  I  was  a  payload  engineer^  a  booster 
systems  easineer.  and  the  officer  in  charge  of  the  mile¬ 
stone  and  sChodulins  program  for  our  nation's  top-priority 
apace  aacellite  system.  And,  I  was  the  PEUT  Officer  for 
the  first  Air  Force  space  project  eitploying  this  manage¬ 
ment  end  control  technique  and  was  the  Configuration  Man¬ 
agement  Officer  for  tho  space  vehicle  most  widely  used  for 
UtAF  end  NASA  space  programs. 

Vhile  serving  in  those  capacities,  I  became  concerned 
with  the  methods  of  evaluating  the  effectiveness  of  tho 
research  and  dovelopeaent  efforts  being  conducted  by  and 
for  tho  Air  Forco  —  tdioro  the  only  r.casure:.*onts  of  perform¬ 
ance  in  sudh  non-profit  orionted  pragrarAs  appeared  to  be 
tho  emount  of  variance  from  targeted  schedules  and  costs 
and  the  dagroa  of  success  c;d'.ibitcd  in  achieving  project 
goals  whan  the  "bird**  was  finally  flcv.'n.  For  these  rca- 
acna^  X  began  to  review  tho  literature  regarding  tho 
maasurement  tochniquos  omplcycd  i^thin  industry  to  evaluate 
raaaacch  and  dovolopsent  offo  rts  — •  thiiUvitag  th.at  in  such  a 
profit-motivated  environment,  more  concrete  means  of  ana- 
lygiag  offactivonosa  might  have  been  developed. 


After  compiling  these  cvalin^tion  methods  for  a  number 
of  years,  I  thought  that  others  might  profit  from  my  euv. 
rasaarOh  of  the  applicable  literature  and  I,  therefore) 
wrote  a  very  brief  report  of  findings,  entitled  "As..^arch 
and  Development:  Appraisal  and  ^aluaticn."  Sub.,ccue:*tly 
published  by  tho  Air  Force  Ins  wa^tkawC  c>«»  •es**ne«>cgy '  s  ucnoca 
of  ^sterns  and  Logistics  in  Kovwmbsr  1970  and  disseminated 
through  tho  Defense  Documontati  on  Ce^rser,  the  report  quiclcly 
gained  widespread  accoptanco  ••  with  requests  for  its  use 
being  received  from  many  different  agencies,  including: 


# 
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(1)  the  U.S,  Atomic  Energy  Commission j  as  well  as  the  Sandia 
Laboratories,  sponsored  by  the  Comr.vission;  (2)  Department 
of  Defense  organizations  such  as  the  Institute  of  Defense 
Analyses,  in  conjunction  with  studies  for  the  BCD's  Weapons 
System  Evaluation  Croup;  the  Air  Force  Technical  Applica¬ 
tions  Center  of  Headquarters ,.  US^iF ;  as  well  as  the  Air 
Force's  Office  of  Scientific  Research,  and  the  Air  Force 
Systems  Command's  Aeronautical  Systems  Division,  Aerospace 
Medical  Division,  and  Aerospace  Research  Laboratories; 

(3)  civilian  agencies  such  as  the  Bell  Telephone  Laborato- 
rieSy  sponsored  by  the  U.S  .A,  Safeguard  Systems  Coirimand; 
Lockheed-Georgia  Company,  sponsored  by  Headquarters,  Aero¬ 
nautical  Systems  Division;  and  Aerojet  General  Corporation, 
sponsored  by  the  U.S.  Army's  San  Francisco  Procurement 
Afioncy;  and  (4)  departr4'icnts  of  management  within  academic 
institutions  such  as  Indiana  University,  Ohio  State  Univer¬ 
sity,  Wright-State  University,  and  many  other  schools. 

The  very  broad  interest  in  the  report  was  typified  by 
the  statement  of  one  requester,  who  said,  "A  docment  that 
coherently  summarizes  appraisal  and  evaluation  techniques 
could  be  of  significant  value  if  used  in  a  conscientiously 
applied  program  of  research  evaluation  and  regular  profes¬ 
sional  revievj."  The  highly  favorable  comments  received 
after  publication  of  the  initial  report  prompted  me  to  pre¬ 
pare  a  more  detailed  documentation  of  my  findings  regarding 
the  methods  and  techniques  of  evaluating  and  measuring  re¬ 
search  and  development  efforts.  Thus,  this  expanded  report 
la  being  published  to  meet  the  apparent  needs  of  many  organ¬ 
isations  interested  in  the  management  of  scientific  and 
tedhnieal  activities.  , 

This  report,  like  the  previous  one,  indicates  that  \:he 
evaluation  of  scientific  performance  in  the  industrial  en¬ 
vironment  is  beset  with  many  of  the  same  problems  that  I 
had  noted  in  the  non-profit  oriented  military  climate. 
Apparently,  no  matter  what  the  R2eD  environment,  the  general 
rule  for  appraising  the  potential  of  proposed  research, 
measuring  the  effectiveness  of  in-progross  activities,  and 
evaluating  the  results  obtained  from  these  efforts  is  simply 
the  follo^ng:  "Use  quantitative  methods,  wherever  appro¬ 
priate,  to  assist  in  making  qualitative  judgments." 

Although  literally  hundreds  of  reforenco  sources  were 
reviewed  in  preparing  this  report,  I  found  several  of 
especial  interest  and  value,  as  may  be  noted  in  referring 
to  my  footnotes  citing  those  sources.  Particularly  meaning¬ 
ful  inputs  were  obtained  from  the  extensive  material 
authored  by  James  B.  Quinn,  as  well  as  by  Raymond  Villers 


J 

i 


and  Walter  McFarland,  I  also  found  a  vcalth  of  rriotorial 
included  in  the  book  on  acialniaterir.s  roooarch  ar,d  dovolc^y 
ment  compiled  by  Charier:  Orth,  Joseoh  Bailey,  and  Francin 
Wolek,  and  in  tho  handbook  or.  industrial  research 

edited  by  Carl  Heyci,  ^^o’,  Jaitos  Bright* a  books  on  tochno- 
logical  innovation  were  vary  infor..'.ative  as  was  Daniel 
Roman*!  text  on  manageriient  of  rasaarch  and  develoratent. 
Indeed,  should  any  of  my  reacicrs  Wa;.sn  ^.o  ciipanQ  tnoar  icncw* 
ledge  of  tho  methods  and  techniques  of  evaluating  research 
and  development  efforts,  any  of  these  cited  authors  along 
with  many  of  those  referenced  in  this  report  v/ould  be  excel* 
lent  sources  of  information  relating  to  the  management  of 
scientific  and  technological  activities,  ^  X 
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Abstract 


Rapidly  advancing  technology  and  accelerating  sci¬ 
entific  progress  have  created  a  dynamic  and  constantly 
changing  industrial  environment  necessitating  ever 
greater  emphasis  on  the  research  and  development  func¬ 
tion  within  all  organizations.  The  measurement  and 
evaluation  of  the  R&D  function  is  difficult  but  none¬ 
theless  necessary,  and  many  different  methods  and 
techniques  have  been  devised  and  used  to  predict  and 
determine  the  results  achieved  from  scientific  and 
technological  endeavor. 

Ranging  across  the  quantitative  to  the  qualitative 
spaetrum  of  methodology,  many  various  means  are  em¬ 
ployed  to  appraise,  measure,  and  evaluate  research 
efforts  and  developmental  activities.  Variables  which 
affect  the  selection  of  a  particular  method  for  analysis 
include  such  considerations  as  the  type  of  research 
agency  involved,  the  level  of  the  research' effort,  and 
the  stage  in  the  developmental  life-cycle  of  the  re- 
aearch  to  be  evaluated.  A  method  which  might  be  quite 
appropriate  under  one  sot  of  variables  might  be  com¬ 
pletely  inapplicable  under  another  sot  of  conditions. 

This  document  reports  many  of  these  means  of  analy¬ 
sis  actually  employed  in  measuring  and  evaluating  re¬ 
search  and  development,  and  it  classifies  these  methods 
and  techniques  as  to  their  applicability  for  pre-appraisal, 
in-process  measurement,  and  post-evaluation.  The  report 
concludes  that,  although  the  quantitative  and  qualitative 
methods  are  at  the  opposite  ends  of  the  measurement 
spectrum,  a  successful  R&D  manager  will  use  quantitative 
means,  whenever  appropriate,  to  assist  him  in  making 
qualitative  Judgments  and  decisions. 
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RESEARCH  AND  DEVELOPMENT 


A  CATEGORIZATION  OF 
THE  METHODS  AND  TECHNIQUES  OF 
MEASURING  INDUSTRIAL  R  §  D  ACTIVITIES 


I.  Introduction 


Ours  has  become  a  dynamic  and  changing  society. 
Rapid  technological  advances  and  new  scientific  dis¬ 
coveries  have  changed  the  face  of  our  economy.  New 
products,  markets,  and  demands  have  altered  the  struc¬ 
ture  of  our  industrial  complex.  Within  this  dynamic 
pattern,  more  and  more  companies  and  industries  have 
come  to  realize  that* their  place  *in  this  structure 
depends  upon  their  adequately  meeting  the  changing 
requirements.  To  improve,  indeed  even  to  maintain, 
their  relative  position  within  the  economy  necessi¬ 
tates  an  emphasis  on  the  technological  aspects  of  their 
operations.  Thus,  increasingly  greater  attention  is 
being  focused  upon  the  research  function  of  their 
organizations  and  the  results  Which  may  be  obtained 
from  industrial  research  and  development. 


II. 


Evaluation  of  Research  Activities 


Industrial  research  activities  have  become  of  ever- 
increasing  importance  to  the  vitality  and  growth  of  our 
economy.  One  authority  has  stated,  "Many  companies  and 
many  industries  realize  today  that  technology  is  an 
important  raw  material  in  their  operations.  .  .  .  The 
effectiveness  of  industrial  research  can  affect  a  com¬ 
pany's  competitive  position,  its  future  growth,  its 
employee  relations,  and  almost  everything  else  connected 
with  what  the  company  does."^  Another  has  said,  "Today's 
research  determines  tomorrow's  growth.  Industry  position, 
profits,  and  research  are  inseparable  in  long  term  plan¬ 
ning  for  corporate  growth.  .  .  .  This  industrial  growth 
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and  development  is  based  upon  research." 

With  these  facts  in  mind,  industry  has  given  increas¬ 
ingly  greater  support  to  research  efforts.  Almost  all 
major  corporations  and  many  industry  associations  have 
established  research  facilities  of  unprecedented  magni¬ 
tude.  Industry  has  funded  much  of  the  research  conducted 
by  private  research  organizations  and  academic  institu¬ 
tions  and  has  established  special  facilities  to  evaluate 
the  potential  commercial  applications  of  the  "spin-off" 
from  various  research  conducted  and  sponsored  by  the 
government.  "Research  has  become  a  mighty  industry  in  its 
own  right  --  perhaps  the  key  industry  of  our  economy."^ 

Huge  sums  are  being  spent  annually  on  research 
endeavor.  Compared  to  the  total  research  expenditures 
of  little  more  than  $100  million  in  1928  and  only  $300 
million  per  year  immediately  prior  to  the  Second  World  War, 
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research  expenditures  in  the  decade  of  the  1960 's  exceeded 
a  rate  of  $12  billion  annually  --  approximately  two  per 
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cent  of  our  gross  national  product  -■  and  they  should  be 
substantially  greater  in  our  present  decade.  Although 
more  than  half  these  expenditures  arc  funded  by  the  Federal 
Government,  funds  contributed  by  industry  amount  to  almost 
two  per  cent  of  the  net  sales  realized  by  companies  per* 
forming  research  and  development.^  Estimates  have  been 
made  that  technological  advances  resulting  from  research 
endeavor  account  for  90  per  cent  of  the  impressive  increase 
in  United  States  productivity  or  output  per  man-hour  while 
only  10  per  cent  is  attributable  to  increases  in  capital 
formation^ 

Raymond  Ewell  of  the  National  Science  Foundation  has 
estimated  that  the  economy  has  earned  a  yield  of  from  100 
to  200  per  cent  a  year  from  its  investment  in  research 

7 

over  the  past  quarter  of  a  century;  James  B.  Quinn  of 
Dartmouth's  Tuck  School  of  Business  Administration  pointed 
out  that  our  annual,  expenditures  on  research  far  exceed 
the  total  amount  spent  from  the  birth  of  the  nation  to 

g 

the  Great  Depression;  and  W.  D.  Seyfried,  Manager  of  Re¬ 
search  for  Humble  Oil  and  Refining  Company,  stated  that  if 

the  present  trend  should  continue,  spending  over  the  next 

g 

five  years  could  be  greater  than  $100  billion.  The 
latter  prediction,  made  in  1963,  has  been  borne  out  by  the 
evidence  of  our  times. 

Although  some  people  discount  the  totals  now  being 
expended  as  being  overstated  because  of  inflationary  re¬ 
search  costs  and  broadening  definitions  of  research  classi¬ 
fications,  others  contend  that  the  expenditures  are  in  fact 
deflated  because  of  the  greatly  increased  scientific 


# 


productivity  which  has  resulted  from  the  use  of  more 
highly  qualified  research  personnel  and  of  greatly 
improved  test  equipment  and  modern  computers. Regard* 
less  of  which  is  true»  the  increase  in  outlays  for 
research  and  development  has  been  tremendous  and  the 
rate  of  expenditures  will  continue  to  accelerate  at  an 
even  greater  pace  in  the  years  to  come. 


Difficulty  of  Evaluating  Performance 

The  vast  amount  of  money  being  spent  annually  on 
research  efforts  emphasizes  one  of  the  most  important 
problems  inherent  to  research  organizations  --  that  of 
determining  some  means  of  evaluating  or  measuxing  the 
performance  of  the  research  function.  Robert  Tannebaum 
of  the  University  of  California's  Institute  of  Industrial 
Relations  described  the  problem  as  follows 

Today,  millions  of  dollars  are  being  spent 
on  research  and  development  work.  Individuals, 
isolated  research  teams,  and  large-scale  labor¬ 
atories  are  working  toward  th^  development  of 
new  ideas  and  methods.  When  such  large  sums  of 
money  and  numbers  of  people  are  concerned,  the 
sound  formulation  of  objectives  and  careful 
evaluation  of  progress  toward  these  objectives 
become  of  increasing  imnortance.  However,  the 
formulation  of  objectives  and  the  evaluation  of 
performance  in  the  research  and  development 
field  have  involved  so  many  intangibles  that  the 
evolution  of  useful  guidelines  has  proven  to  be 
most  difficult. 

This  position  is  substantiated  by  Edward  B.  Roberts, 

1 2 

in  his  book.  The  Dynamics  of  Research  and  Development; 

These  intangible,  imprecise,  and  uncertain 
factors  prevent  charting  the  accomplishment  of 
a  significant  research  and  development  undertaking. 
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.  .  .  Attempts  to  apply  standard  control  and 
evaluation  techniques  to  the  research  program 
have  proved  very  disappointing.  .  .  .  The 
problems  involving  measuring  and  responding  to 
progress  in  a  research  and  development  project 
.  .  .  arise  out  of  the  inapplicability  to 
research  and  development  of  existing  control 
methods.  .  .  .  Project  managers  continously  try 
to  state  what  progress  has  been  made,  but  no 
necessarily  relevant  measurements  are  available 
to  verify  or  deny  such  statements. 

This  appears  to  be  the  crux  of  the  problem.  "By  its 
very  nature,  research  is  the  business  of  getting  disorgan- 
ized."  Because  of  this,  the  research  function  (and  the 
effects  achieved  therefrom)  tends  to  defy  measurement. 

The  imperspicuous  and  chimerical  nature  of  measuring  re¬ 
search  performance  is  cited  in  virtual  unaniminity  by  all 
authorities  on  the  subject. 

Herbert  A.  Shepard,  in  describing  the  dilemmas  of 

industrial  research  and  the  inapplicability  of  traditional 
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methods  of  measurment,  stated: 

Probably  no  other  branch  of  industrial 
activity  causes  so  much  mana'gerial  unrest  and 
uncertainty  as  does  research  and  development. 

The  concern  is  due  largely  to  the  enormous  diffi¬ 
culty  of  evaluating  the  laboratory's  actual  and 
potential  economic  contributions  to  the  firm. 

In  a  study  for  the  National  Industrial  Conference 
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Board,  Nestor  Terleckyj  and  Harriet  Halper  found  that: 

Although  some  limited  information  exists 
suggesting  that  the  relationship  between  research 
and  development,  on  the  one  hand,  and  growth, 
productivity,  and  profits  of  industries  and  com¬ 
panies,  on  the  other,  is  positive,  no  quantitative 
measures  of  R§D  payoff  have  as  yet  been  developed. 

In  view  of  the  diversity  of  R§D  operations 
and  the  lack  of  any  identifiable  product  unit,  it 
should  be  clear  that  no  general  measure  of  output 
of  R^O  can  be  devised  at  present. 


In  discussing  the  general  uncertainty  of  the  inci¬ 
dence  of  exploitable  research  findings,  Ralph  E.  Burgess, 
a  leading  consultant,  noted,  "This  vagueness  makes  it 
difficult  to  measure  the  gains  from  past  research,  to 
determine  the  proper  scope  of  current  effort,  to  estab¬ 
lish  a  cut-off  point  for  projects  that  appear  fruitless, 
and  to  anticipate  future  gains. After  a  detailed 
study  of  management  controls  in  industrial  research 
organizations,  Robert  N.  Anthony  concluded,  "There 
seems  no  way  of  measuring  quantitatively  the  performance 
of  a  research  laboratory.  Competent  persons  are  general- 

ly  reluctant  to  rate  the  performance  of  various  labora- 
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tories  on  any  basis."  His  findings  were  further  sub¬ 
stantiated  by  the  results  of  a  National  Science  Foundation 
survey  which  found  that  less  than  a  quarter  of  200 

companies  interviewed  reported  any  formal  methods  of  esti- 
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mating  research  results. 

Typically,  research  results  are  difficult  to  predict, 
projections  of  intended  results  are  usually  of  insuffi¬ 
cient  accuracy  to  serve  as  standards  of  performance,  and 
actual  results  are  frequently  quite  different  from  those 
planned.  Often  the  only  results  obtained  from  many  re¬ 
search  programs  are  an  increased  knowledge  of  the  subject 
area,  the  formulation  of  new  ideas,  and  an  improvement  of 
research  techniques  --  all  admittedly  of  an  intangible 
nature.  Further,  the  fruits  of  research  may  not  be  real¬ 
ized  until  some  distant  future  date.  Because  the  parti¬ 
cular  application  of  some  given  research  results  may  lie 
dormant  for  decades,  the  full  economic  impact  and  return 
on  investment  may  not  be  realized  until  many  years  after 


the  research  program  was  performed.  Summing  up  these 

problems,  James  B.  Quinn  said,  "Comparing  intangible 

results  with  inaccurate  forecasts  can  only  yield  nebulous 
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conclusions." 

Necessity  for  .Measuring  Effectiveness 

Although  there  is  almost  universal  agreement  as  to 
the  impossibility  of  measuring  or  evaluating  the  research 
function,  all  authorities  on  the  subject  also  agree  as  to 
the  necessity  of  doing  so  simply  because  of  the  rapidly 
growing  expenditures  for  research  and  because  of  the  fact 
that,  if  companies  are  to  continue  to  invest  huge  sums  in 
research  and  assume  the  risks  of  supporting  sizeable  re- 
search  programs,  they  must  know  whether  their  funds  are 
being  used  effectively  and  whether  the  research  output 
justifies  the  cost.^^ 

More  specifically,  the  company  .management  wants  to 
know  whether  the  research  investme/.ts  arc  worthwhile  at 
all,  how  the  returns  compare  with  those  which  could  be 
realized  from  alternative  investments,  and  whether  the 
effort  is  maximumly  effective.  The  .ma.nagcment  desires 
to  judge  the  past  performance  or  output  of  research  in  a 
manner  which  will  provide  guidance  for  future  research 
and  operating  effort. 

Going  even  further,  the  research  director  of  one  of 
the  nation's  largest  companies  contends  that  research  it¬ 
self  is  of  little  value  if  its  results  cannot  be  measured: 

In  the  professional  work  of  the  research 
^  scientist,  measurement  is  an  essential  activity. 

”  A  theory  of  nature  is  not  truly  descriptive  until 
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it  becomes  quantitative,  nor  is  a  scientific  ob¬ 
servation  fully  meaningful  until  numbers  can  be 
assigned  with  the  phenomena  under  study.  [Meas¬ 
urement  of  the  functional  work  of  scientists]... 
are  of  necessity  if  scientific  research  is  to 
grow  in  magnitude  and  contribution  to  American 
industry. 

The  somewhat  excessive  number  of  quotations  given 
in  the  preceding  paragraphs  represents  but  a  few  of  many 
that  could  have  been  cited  to  illustrate  the  generally 
broad  consensus  held  by  most  authorities  on  the  research 
and  development  function:  first,  that  the  evaluation 
and  measurement  of  research  activities  is  complex  and 
difficult;  and  second,  that  it  is  nevertheless  necessary. 
Granted  these  two  points,  the  question  arises  as  to  how 
the  effectiveness  of  research  performance  can  be  measured. 
This  brings  in  a  variety  of  other  factors  which  must  be 
considered,  such  as  the  nature,  the  types,  the  levels, 
and  the  stages  of  research  to  be  evaluated,  interwoven 
with  considerations  of  the  most  applicable  methods  of 
evaluation  and  the  effects  of  evaluation  upon  the  research 
personnel  involved.  Although  the  'primary  focus  of  this 
paper  is  on  the  evaluation  of  industrial  research  and 
development,  these  broader  areas  will  first  be  addressed 
before  returning  to  the  main  topic  of  interest. 


Nature  of  Research 

Research  is  systematic  investigation  designed  to 

seek  answers  to  questions  or  solutions  to  problems. 

Although  the  word  itself  suggest  a  re-examination  or  a 

repeated  search,  the  following  provides  a  better  under- 
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standing  of  its  meaning: 


What,  then,  is  research?  It  is  not  mere 
RE-search  (although  plenty  of  that  has  to  be  done); 
it  is  more  than  hashing  over  the  work  of  others. 

It  is  re-SEARCH,  fresh  inquiry  into  and  seeking 
for  something  new  and  perhaps  better,  despite  the 
fact  that  "better"  is  a  value-word,  and  therefore 
tabu  in  some  academic  mores.  Here  originality 
comes  in. 

Research  is  purposeful,  objective,  and  systematic 
analysis,  distinguishable  from  informal  observation  by  its 
conceptual  methodology.  The  research  method  is  purposive, 
for  it  is  the  planned  seeking  of  answers  to  specific 
questions  or  problems.  It  is  objective,  for  it  minimizes 
bias  and  prejudice  in  its  processes.  It  is  systematic, 
for  it  is  carried  out  step-by-step  to  follow  a  well-defined 
plan.  And  it  is  analytical,  for  it  reports  conclusions 
based  upon  the  principles  of  logic  rather  than  a  summary  of 
unclassified  facts. 

Levels  of  Research 

The  levels  of  research  may  be  broadly  categorized  as 
pure,  basic,  applied,  and  developmental  research.  Pure 
research  is  fundamental  inquiry.  It  is  largely  an  end  in 
itself,  gathering  facts  for  the  si.mple  reason  that  facts 
are  worth  gathering,  and  unconcerned  whether  or  not  the 
facts  are  of  immediate  use  in  solving  an  existing  problem. 
Pure  research  simply  seeks  to  add  to  the  accumulation  of 
knowledge  and  understanding  by  providing  answers  to  general 
inquiries.  Basic  research,  on  the  other  hand,  investigates 
broad  questions,  and  seeks  to  provide  answers  that  may  be 
applicable  to  many  and  varied  practical  problems.  Although 
its  findings  are  generalized,  they  may  be  practically 


applied  to  various  situations.  Applied  research  differs 
from  basic  research  only  in  its  specificity  of  investiga¬ 
tion.  It  is  used  to  study  a  specific  question  or  problem 
area  (which  may  be  either  simple  or  complex)  in  order  to 
provide  solutions  which  have  immediate  and  practical 
applicability.  Developmental  research  is  the  last  stage 
in  the  chain  of  incorporating  more  basic  research  findings 
into  operational  practicality.  Its  purpose  is  the  resolu¬ 
tion  of  any  final  probelms  standing  in  the  way  of  actual 
application  of  the  research  findings  into  a  usable  product. 

There  is  no  clear  line  of  demarcation  between  the 
various  levels  of  research,  for  one  may  lead  to  or  blend 
in  with  another.  What  is  defined  as  pure  research  may  in 
actuality  be  basic,  and  what  is  designated  as  basic  may 
really  be  applied  research.  Especially  hazy  is  the  bound¬ 
ary  between  applied  and  developmental  research,  for  these 
two  are  often  considered  as  synonomous.  The  problems  of 
classification  of  research  levels  are  illustrated  in  the 
following  two  statements:  , 

We  define  basic  research  as  the  search  for 
new  phenomena  and  relationships  in  nature  and 
applied  research  as  the  exploration  for  and  crea¬ 
tion  of  product  possibilities  out  of  science  and 
technology.  We  are  excluding  development  and 
engineering  from  our  considerations  because  we 
assume  that  when  a  project  reaches  the  development 
stage,  the  worth  of  the  research  has  already  been 
established.  We  must  remember,  however,  that 
basic  research,  applied  research,  and  development 
and  engineering  form  a  continuum.  They  are  often 
carried  on  simultaneously  in  such  a  manner  as  to 
prevent  simple  classification  of  a  project. 

The  phase  which  occurs  earliest  in  the  devel¬ 
opment  cycle  is  pure  or  basic  research.  Its 


result  is  creation  of  ncxv  knowledge,  which,  in 
the  physical  sciences,  is  concerned  with  struc¬ 
tural  and  dynaaic  properties  of  aattcr.  It  is  the 
process  of  creating  aatheaatical  equations  and/or 
experimental  procedures  which  explain  observable 
(nan* made  or  nature 'made)  phenomena. 

Applied  research,  which  follows,  is  the  appli¬ 
cation  of  knowledge  to  the  solution  of  previously 
unsolved  (usually  generic}  proble.ms.  It  can 
either  bo  device-oriented  ...  or  it  can  be  tech¬ 
nique-oriented.  .  .  . 

The  fact  that  the  degree  of  uncertainty  (of 
the  outcome  of  a  proposed  project}  can  vary  over  a 
vide  spectrum  gives  rise  to  a  .major  difficulty  in 
defining  clear-cut  separation  between  applied  re¬ 
search  and  activities  which  precede  and  follow  it. 

At  one  extreme,  it  is  not  uncor..mon  to  find  applied 
research  proposals  whose  outcome  is  so  uncertain 
as  to  require,  in  fact,  creation  of  new  knowledge 
and  which  should,  therefore,  be  reco^nited  as  basic 
research.  .  .  .  Confu.<(ior.  as  to  the  true  nature  of 
applied  research  occurs  most  frequently  at  the 
other  end  of  the  spectrum  of  uncertainty,  where  it 
may  be  confused  with  a  later  stage  in  the  product 
devolqosent  cycle,  which  is  called  product  develop¬ 
ment.*" 

These  two  statements  the  first  by  G.  K.  Teal  and 
the  second  by  I.  K.  Ansoff,  both  rocognited  authorities  on 
the  subject  of  research  --  clearly  illustrate  the  problc.T. 
involved  in  trying  to  classify  the  levels  of  research 

effort. 


Catofiorizine  the  Levels 

The  degree  of  uncertainty  of  the  results  of  research 
appears  to  be  the  basic  criterion  for  categorizing  the 
level  of  research  effort.  At  one  end  of  the  spectrum  of 
uncertainty  is  fundamental  research  and  at  the  ot.hcr  end 
is  developmental  research.  Based  on  this  spectrum,  James 
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R.  Bright  has  identified  technical  innovations  as  falling 
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into  one  of  the  four  following  categories: 

Routine  Improvements  .  .  .  are  minor  engi¬ 
neering  changes  in  which  the  technical  knowledge 
for  design,  implementation,  and  application  are 
known  and  predictable.  No  research  is  required; 
nor  is  exploratory  development  needed. 

.Major  Advances  .  .  .  are  substantial  changes 
in  which  the  technological  direction  and  needs  are 
within  the  "state  of  the  art,"  although  not  pre¬ 
viously  performed.  .  .  .  The  necessary  technology 
is  identifiable  and  seems  to  be  achievable.  The 
technical  program  can  be  designed  and  planned  with 
some  certainty.  Little  new  scientific  knowledge 
is  required,  although  engineering  development  will 
be  needed  to  resolve  the  design  details. 

Technological  Breakthrough  .  .  .  involves  a 
radical  change  or  improvement  that  requires  new 
scientific  knowledge  and  consequent  development 
work.  Applied  and,  possibly,  basic  research  is 
necessary.  The  technological  answers  seem  to  be 
feasible,  but  are,  for  the  moment,  unconfirmed  or 
unknown • 

"Blue-skyV  Projects  .  .  .  arc  technological 
proposals  in  which  the  means  of  accomplishment  or 
the  end  results  are  largely  Unknown.  Substantial 
new  scientific  knowledge  and  consequent  engineering 
knowledge  is  required.  Basic  research  and  applied 
research  is  indicated.  There  is  generally  a  com¬ 
pounding  of  many  technological  uncertainties  in 
knowledge,  equipment  needs,  and  results.  Therefore 
predictability  is  very  low. 

Yet  another  author  has  segregated  industrial  research 
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activities  into  the  following  project  categories: 

Pure  research,  i.e.,  direct  research  or  ex¬ 
perimentation  on  general  problems  having  no  parti¬ 
cular  connection  with  the  various  products  currently 
being  manufactured  by  the  plant. 


Projects  directing  experimental  or  develop¬ 
mental  effort  toward  the  creation  of  now  processes 
or  new  products  or  groups  of  products  currently 
being  manufactured  by  the  plant.  .  . 

Projects  directing  experimental  or  develop¬ 
mental  effort  toward  any  improvement  to  a  specific 
product  already  being  manufactured  by  the  plant  or 
an  improvement  of  an  existing  process.  .  . 

All  further  work  beyond  the  developmental 
stage  necessary  to  get  a  new  product,  model,  or 
item  of  equipment  ready  for  normal  production  and 

sale . 

Projects  for  the  purpose  of  designing  and 
constructing  new  types  of  equipment  or  improvements 
to  existing  equipment  which  shall  be  used  in  our 
manufacturing  processes  and  which  will  effect  a 
change  in  any  existing  process  in  the  plant. 


Interrelationships 

Although  the  majority  of  present  research  expenditures 
are  made  in  the  areas  of  applied  and  developmental  re¬ 
search,  it  is  important  to  remember  that  all  levels  or 
categories  of  research  are  needed,  since  each  contributes 
to  the  improvement  of  understanding  and  practice.  Though 
more  importance  may  be  placed  on  one  type  of  research  over 
another  in  a  given  time  or  place,  advancement  in  technology 
can  easily  be  retarded  if  one  type  is  emphasized  to  the 
detriment  of  another.  All  are  interrelated  and  support  one 
another:  pure  research  often  reveals  new  facets  of  under¬ 
standing  that  have  important  implications  to  the  solution 
of  practical  problems,  and  applied  research  frequently 
provides  essential  data  which  opens  up  new  vistas  for  pure 
research  inquiries.  Thus  all  levels  of  research  are  needed 
and  must  be  jointly  conducted  for  progress  to  be  achieved.*^ 


Implication  for  Evaluation 


Assuming  that  the  spectrum  of  uncertainty  ranges 
from  a  great  deal  of  uncertainty  at  the  lower  end  of  the 
spectrum  to  a  fairly  high  degree  of  certainty  at  the 
upper  end,  then  the  principal  implication  for  evaluation 
of  research  performance  lies  in  the  fact  that  research 
results  may  be  measured  with  some  degree  of  confidence 
at  the  upper  end  of  the  spectrum  (i.e.,  in  the  areas  of 
greater  certainty)  whereas  at  the  lower  end  (i.e.,  in 
the  areas  of  greater  uncertainty)  the  measurement  of 
research  effectiveness  becomes  progressively  more  diffi¬ 
cult. 

Thus,  developmental  research,  falling  within  the 
upper  end  of  the  spectrum  of  uncertainty,  is  closely  con¬ 
cerned  with  producing  a  product  whose  output  may  be 
reasonably  well  compared  to  the  inputs  required  to  produce 
the  product,  For  this  level  of  research,  more  quantita¬ 
tive  means  of  evaluation  may  be  effectively  employed.  In 
the  absence  of  a  profit-motive  (s'uch  as  the  research  per¬ 
formed  by  or  for  governmental  agencies),  the  item  which 
is  produced  may  be  evaluated  against  its  design  specifi¬ 
cations  to  determine  the  effectiveness  of  the  research 
effort.  When  the  developmental  effort  leads  to  a  good 
produced  for  sale,  the  resultant  profits  may  be  further 
quantitatively  measured  against  the  development  costs  to 
determine  research  effectiveness.  On  the  other  hand,  the 
fundamental  levels  of  research,  falling  within  the  lower 
end  of  the  spectrum  of  uncertainty,  remain  more  difficult 
to  measure,  and  qualitative  or  subjective  yardsticks  must 
be  used  to  evaluate  how  well  the  research  effort  has 
succeeded. 
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Types  of  Research 

Many  various  types  of  research  effort  are  conducted 
by  many  differer4t  agencies  and  organizations,  but  nost 
research  endeavors  could  be  included  under  the  following 
categories:  research  conducted  by  or  for  the  govern:r.ent 
on  defense  needs  and  on  health,  welfare,  enviro  r4ir«o  I*  tal , 
and  related  programs;  research  conducted  by  business 
enterprises  to  maintain  or  advance  their  competitive  pcsi' 
tion  or  prestige  within  their  given  industry;  and  research 
conducted  by  universities,  r4cn*profit  institutions,  and 
private  research  organizations  for  any  of  these  purposes 
just  cited  or,  simply,  to  increase  and  broaden  basic  know¬ 
ledge. 


Government 

Approximately  half  of  the  total  research  expenditures 
within  the  nation  are  made  by  the  government.  A  large 
proportion  of  these  government  expenditures  are  primarily 

4 

for  defense  and  space  research,  but  an  increasingly 
greater  amount  goes  to  other  areas  of  research,  such  as 
atomic  energy,  food  and  drug,  agriculture  and  forestry, 
natural  resources,  public  health,  and  similar  areas  of 
domestic  concern.  Although  much  of  the  government -financed 
research  is  performed  within  government  laboratories,  a 
great  deal  is  done  by  non-profit  research  institutions  and 
by  colleges  and  universities.  In  fact,  one  survey  revealed 
that  almost  three  quarters  of  the  fu.nding  for  research  by 
academic  institutions  came  from  the  I^cderal  government 


Although  student  protests  within  recent  years  against 
the  involvement  of  educational  institutions  in  government- 
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sponsored  research  pro^jrams  have  resulted  in  the  curtail¬ 
ment  of  such  research  by  some  universities,  it  is  still 
expected  that  most  of  the  institutional  research  efforts 
will  continue  to  be  government -funded  within  the  foresee¬ 
able  future. 

Academic  Institutions 

One  study  has  revealed  that  colleges  and  universities 
are  expending  nearly  $1  billion  per  year  on  research  efforts 
of  various  types.  Most  of  the  pure  and  basic  research 
within  the  United  States  is  conducted  by  academic  institu¬ 
tions,  with  about  half  of  their  effort  being  devoted  to  the 
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pursuit  of  fundamental  knowledge.  Many  institutions  have 
established  bureaus  of  research  that  conduct  applied  and 
operational  research  for  individual  firms  within  industry 
and  for  trade  associations.  A  portion  of  their  income  from 
such  projects  is  used  to  finance  pure  and  basic  research 
conducted  by  individual  faculty  members  of  various  depart¬ 
ments.  In  more  recent  years,  extensive  use  has  been  made 
of  research  organizations  established  separately  from  the 
academic  institutions,  either  as  mere  extensions  of  the 
institutions  themselves,  or  as  quasi -independent  bodies  or, 
even,  as  separate  legal  entities,  receiving  funds  primarily 
by  selling  their  services  to  the  government,  business  enter¬ 
prises,  and  foundations.^^ 

Private  Organizations 

Private  research  organizations,  ranging  from  non-profit 
research  institutes  to  research  consultant  firms,  expend 
approximately  $100  million  per  year  on  research  efforts. 
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according  to  one  estimate.  [Although  often  included  in 
this  category  of  research  organizations  are  the  various 
business  and  industry  associations  that  conduct  some 
research  projects,  they  are  more  appropriately  listed  under 
industry  research  efforts.] 

These  private  organizations,  whether  they  be  non-profit 
or  commercial,  depend  upon  clients  for  their  business,  and 
they  sell  their  services  to  governmental  agencies,  educa¬ 
tional  institutions,  and  business  firms.  Their  orientation 
may  be  toward  fundamental  scientific  research;  they  may  be 
oriented  toward  applied  and  operational  research;  or  their 
concern  may  be  with  national  and  economic  research.  However, 
most  of  the  endeavor  of  the  commercial  laboratories  is  of 
the  applied  and  developmental  research  level,  while  that  of 
the  non-profit  institutes  lies  at  the  other  end  of  the 
research  spectrum. 

Industry 

I 

Research  expenditures  by  industry  account  for  nearly 
half  of  the  total  research  expenditures  within  the  nation. 
Although  something  like  95  per  cent  of  this  funding  goes  to 
applied  research  and  development,  the  remaining  amount 
expended  on  basic  and  fundamental  research  nearly  equals  the 
amount  funded  by  academic  institutions  for  the  same  purpose. 
Although  much  of  this  basic  research  within  industry  is  per¬ 
formed  in  company  research  organizations,  a  significant 
amount  of  the  industry  research  funding  goes  to  research 
conducted  by  employer  associations,  private  organizations, 

7  A 

and  academic  institutions. 
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Implications  for  Evaluation 

The  principal  implication^  concerning  the  evaluation 
of  research  effectiveness  of  the  types  of  agencies  which 
conduct  research  is  essentially  the  same  as  that  arising 
from  the  spectrum  of  uncertainty.  The  more  basic  the 
research  performed  by  academic  institutions,  some  govern¬ 
ment  laboratories,  and  a  few  business  enterprises,  the 
more  subjective  or  qualitative  must  be  the  means  of  meas¬ 
urement  of  these  research  efforts.  On  the  other  hand, 
the  applied  research  and  development  efforts,  falling 
within  the  more  certain  end  of  the  spectrum  of  uncertain¬ 
ty,  may  be  guaged  by  more  quantitative  methods  of  evalua¬ 
tion. 

Stages  of  Research 

The  stages  of  research  are  what  has  often  been  called 
the  developmental  cycle  --  running  from  initial  consider¬ 
ation  of  proposed  research  through  in-process  activities 
to  the  completion  or  termination  of  the  project.  Many 
tend  to  confuse  the  stages  of  research  with  the  levels  of 
research,  thinking  that  the  proposed  research  stage  is 
akin  to  the  fundamental  research  level,  that  the  in-proccss 
stage  is  comparable  to  the  basic  or  applied  research  level, 
and  that  the  completion  stage  is  similar  to  the  develop¬ 
mental  level.  While  fundamental  research  often  leads  to 
the  development  of  products,  this  should  not  be  confused 
with  the  life-cycle  or  stages  of  progress  of  research 
efforts  in  the  developmental  cycle.  Any  research  project- - 
be  it  fundamental  or  developmental --usually  goes  through 


a  cycle  of  activity  stages,  initially  with 

consideration  of  the  proposed  research  ar4d  ultimately 
concluding  in  some  sort  of  research  results. 

To  determine  how  effective  is  the  research  effort 
of  organizations,  some  means  of  evaluatio.^.  must  be 
employed  to  measure  the  research  program  at  the  various 
stages  of  their  developmental  life- cycles.  .\s  cue 
authority  has  stated,  "Evaluation  of  ar.y  activity  gen¬ 
erally  implies  that  there  is  a  predetermined  set  of 
values  or  philosophy  that  will  serve  as  a  criterion  in 

making  the  evaluation.  And  evaluating  a  product  research 
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program  is  no  exception."  Thus,  techniques  for  re¬ 
search  evaluation  must  relate  to  these  predetermined 
criteria. 

Implications  for  Evaluation 

Techniques  for  pre-appraising  proposed  projects  are 
primarily  concerned  with  establishing  a  set  of  values 
and  criteria  to  assist  in  the  evaluation  and  selection  o 
of  alternative  research  proposals  and  against  which  sub¬ 
sequent  effort  and  results  may  be  measured  once  a  project 
is  inaugurated.  Techniques  for  evaluating  in-process 
activities  normally  attempt  to  measure  the  current  status 
or  degree  of  completion  of  research  efforts  against 
criteria  initially  planned  for  the  various  points  in  the 
developmental  cycle.  And,  techniques  for  post -appraising 
projects  try  to  evaluate  the  effectiveness  of  research 
results  viewed  in  the  light  of  the  pre-established  goals 
and  criteria.  These  techniques  of  evaluation  range 
across  the  entire  spectrum  of  methodology  --  from  the 
qualitative  to  the  quantitative  ends  of  the  spectrum. 
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Methods  of  Evaluation 

Although  there  are  many  different  methods  of  evalu¬ 
ating  research  potential,  performance,  and  effectiveness, 
they  may  be  grouped  into  three  basic  categories  --  quanti'- 
tative,  qualitative,  and  some  combination  of  the  two. 

Quantitative  Methods 

The  quantitative  methods  typically  use  mathematical 
equations  or  models  coupled  with  dollar  evaluation  (when 
available)  to  arrive  at  some  figure  which  describes  the 
merit  or  effectiveness  of  research  efforts;  "...they 
range  all  the  way  from  simple  formulas  -*  which  use  com¬ 
pany  profits  for  a  given  period  of  time  as  the  numerator 
and  research  costs  as  the  denominator  --to  more  sophis¬ 
ticated  approaches  involving  present  worth  and  discounted 
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cash  flow."  They  attempt  to  measure  the  contribution 

of  the  research  program  by  means  of  mathematical  equations 

relating  to  the  profits  from  research  created,  improved, 

or  cost  reduced  products,  techniques,  processes,  materials, 

or,  even,  good-will,  and  to  the  savings  engendered  from 
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costs  avoided  or  income  received  from  royalties.  These 

are  compared  with  the  costs  of  operating  the  research 

function,  and  these  comparisons,  in  turn,  are  often  com- 
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pared  to  similar  data  on  other  research  organizations. 

Proponents  of  the  quantitative  methods  of  analysis 
claim  that  sound,  objective  formulas  numerically  present 
the  most  accurate  possible  estimates  of  research  costs  and 
returns  and  that,  in  simplifying  the  details  of  research 
programs  into  easily  understood  dollar  figures  and  ratios, 
they  help  gain  the  support  of  top  management  for  research 
efforts. 


Opponents  of  quantitative  measurement  devices  cite 
many  difficulties  in  their  use:  It  is  often  impossible 

to  correlate  costs  with  performance  and  one  research 
organization  with  another,  because  of  differences  in 
accounting  practices,  size  and  functions  performed, 
terminology  used,  or,  simply,  the  absence  of  valid  data. 
Quantitative  methods  can,  at  best,  be  used  only  to  a 
limited  degree  for  appraising  research  projects;  often 
the  data  needed  for  the  formulas  are  of  such  questionable 
accuracy,  or  unavailable,  that  the  results  are  of  little 
value.  The  time-consuming  characteristics  of  some  of  the 
more  sophisticated  techniques  often  outweigh  any  advan¬ 
tages  gained  by  their  use,  and  the  rigid  application  of 
inflexible  formulas  frequently  gives  misleading  results 
which  careful  judgment  would  not  have  done.  Since  the 
elements  of  the  formulas  themselves  must  be  determined  by 
individual  judgment  initially,  then  it  would  seem  that 
judgment  alone  is  adequate  without  resorting  to  the  use 
of  complex  and  time-consuming  equations. 

The  opposing  views  toward  the  use  of  quantitative 
methods  of  evaluation  are  not  totally  irreconcilable. 
Within  each  group  of  statements  there  may  be  found  some 
items  of  merit  which,  when  considered  in  relation  to  the 
variable  dimensions  described  by  the  levels  of  research 
effort  and  the  stages  in  the  developmental  cycle,  might 
be  summarized  as  follows:  The  quantitative  methods  are 

best  used  in  the  applied  research  and  developmental  end 
of  the  spectrum  of  uncertainty.  Their  use  for  evaluating 
pure  and  basic  research  efforts  is  of  questionable  value, 
as  is  their  use  in  measuring  the  performance  of  non-pro fi 


motivated  research  endeavor.  And,  they  are  normally  of 
greater  value  in  the  post-appraisal  stages  of  research 
evaluation  than  in  the  pre-appraisal  stages  where  greater 
uncertainty  of  data  is  prevalent. 

Qualitative  Methods 

Because  of  the  weaknesses  cited  in  the  use  of  quanti¬ 
tative  methods  of  evaluating  research  efforts,  many 
organizations  heavily  rely  on  qualitative  measurements  or 
subjective  judgment.  Qualitative  evaluation  is  usually 
the  result  of  a  composite  judgment  of  qualified  and  re¬ 
sponsible  management  and  research  personnel  arrived  at 
through  logical  but  nonmathematical  procedures  and  devices. 
Such  procedures  normally  consist  of  a  tiered  or  pyramidal 
layer  of  appraisals  by  all  concerned  with  the  research 
function.  Judgments,  based  on  personal  experience  and 
"feeling  for  the  situation,"  arrived  at  through  day-to-day 
technical  management  activities  and  periodic  formal 
reviews,  are  used  to  evaluate  the , research  effort.  Often, 
certain  serai-quantitative  yardsticks,  such  as  comparisons 
with  the  results  obtained  by  competitors  or  by  others 
within  the  organization  or  with  the  relationships  of 
achievements  to  stated  objectives,  are  employed  to  sup¬ 
port  qualitative  appraisals. 

The  advantage  of  qualitative  methods  is  that  they 
rely  on  the  judgment  of  experienced  and  responsible  re¬ 
search  personnel  and  thereby  avoid  some  of  the  pitfalls 
of  the  more  inflexible  quantitative  measures.  However, 
if  qualitative  analysts  completely  ignore  the  use  of 


appropriate  nuacrical  data  which  could  be  usex'ul  ir».  cval 
uating  research  effectiveness,  they  bypass  i.T.portaat 
tools  which  could  be  used  to  sharpen  their  subjective 
analyses.  In  fact,  by  nerely  foraulatin”  subjective 
judfinents,  qualitativists  wei^h  n;any  of  the  factors  whic 
are  included  in  quantitative  analysis.*"^ 


In  sumnary,  the  qualitative  itetheds  are  best  suited 
to  evaluating  the  fundamental  end  of  the  research  spec- 
trun.  They  can  be  used  to  advantage  in  measuring  the 
quality  of  research  results,  the  degree  of  research  effi 
ciency,  the  results  of  long-range  research,  the  intangib 
products  of  research  such  as  publications,  arid  the  non¬ 
profit  BOtivated  type  of  research  such  as  that  conducted 
by  governaent  laboratories.  They  may  also  be  of  greater 
value  than  quantitative  methods  in  the  pre -appraisal 
stages  of  the  research  developmental  cycle. 


Conbination  >tethods 

Obviously,  the  quantitative  vbrsus  the  qualitative 
Methods  are  at  the  opposite  ends  of  the  measurement 
spectrum.  Variations  of  one  or  the  other,  or  combina¬ 
tions  of  the  two,  are  normally  employed.  The  use  of  o.ne 
■ay  be  tempered  by  the  use  of  the  other.  The  somewhat 
obvious  advantages  of  the  use  of  combination  methods  of 
analysis  will  be  further  discussed  later  in  this  paper. 


III.  Evaluation  of  Industrial  Research  Endeavor 


In  reviewing  the  literature  relating  to  the  evalua¬ 
tion  of  industrial  research  efforts,  it  was  noted  that 
statements  were  often  made  that,  "our  company  uses  the 
XXXXX  method  for  appraising  its  research  programs,"  or 
"we've  found  the  best  technique  to  evaluate  our  R^D 
is  XXXXX."  Seldomly  did  these  authors  state  where  in 
the  developmental  cycle  was  their  particular  method  of 
evaluation  employed,  nor  for  that  matter  at  what  level 
of  research  effort  was  the  means  of  measurement  used. 

In  an  attempt  to  create  some  degree  of  order  in  evalua¬ 
ting  the  different  methods  and  techniques  employed  for 
ey^uating  of  industrial  research  efforts,  these  various 
means  have  been  classified  as  to  their  applicable  use 
at  various  points  in  the  developmental  life-cycle;  i.e., 
the  methods  have  been  classified  as  to  their  applica¬ 
bility  for  pre-appraisal,  in-proc;ess  measurement,  and 
post-evaluation. 

Pre-Appraisal 

Widespread  use  of  both  quantitative  and  qualitative 
methods  of  evaluation  are  found  for  pro -appraising  alter¬ 
native  research  proposals. 

Some  of  the  more  qualitative  methods  are  concerned 

with  such  fundamentals  as  linking  proposed  research  to 

company  goals.  For  instance,  the  assistant  vice-president 

of  research  and  engineering  for  one  of  the  nation's  most 
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progressive  electronics  firms  has  stated: 
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The  heart  of  the  problem  of  selecting 
worthwhile  research  projects  in  an  industrial 
laboratory  is  matching  research  to  company 
goals.  Company  goals  should  take  advantage 
of  economic,  social,  and  technological  trends 
and  be  based  on  considerations  such  as  manage¬ 
ment's  aspirations  and  the  company's  maturity, 
economic  condition,  and  capabilities  in  all 
aspects  of  creating,  making,  and  selling 
products .... 

To  be  considered  worthwhile,  a  project 
must  have  the  following  general  characteristics: 

1.  It  generates  good  industrial  tech¬ 
nology. 

2.  It  produces  new  product  ideas  or  ideas 
for  product  improvement. 

3.  It  enhances  company  prestige  in  the 
scientific  community. 

4.  It  has  a  reasonable  probability  of 
success . 

5.  It  challenges,  extends,  and  trains 
technical  personnel  project  leaders. 

6.  It  has  the  enthusiastic  support  of  a 
project  manager. 

Another  company  vice-president  for  research  and 

technology  established  the  .following  criteria  for  consid- 
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eration  in  evaluating  research  proposals: 

Criteria:  Will  the  results  of  the  project, 
assuming  success  in  the  research  phases,  pro- 

duce-- 

1.  A  saving  or  increase  in  profits  com¬ 
mensurate  with  the  anticipated  cost  of  research 
and  development? 

2.  An  improvement  in  quality  essential  to 
the  maintenance  of  the  company's  competitive 
position? 

3.  An  increase  in  the  volume  of  business 
on  currently  produced  items  commensurate  with 
the  cost  of  the  research  and  development? 

4.  A  substantial  improvement  in  consumer, 
customer,  or  government  relations? 

5.  A  new  product? 


♦ 


Will  it  bc-- 


If  a  new  product: 

1.  One  that  fits  lo^^ically  into  the  com¬ 
pany's  existing  lines  of  business? 

2.  One  for  which  there  is  an  anticipated 
market  demand? 

5.  One  for  which  raw  materials  are 
available? 

4.  One  which  can  be  manufactured  and  sold 
profitably  over  at  least  a  normal  payout 
period? 

5.  One  that  can  be  produced  with  existing 
plant  facilities;  or,  where  new  facilities  are 
required,  one  where  the  capital  investment  and 
resulting  depreciation  charges  are  not  excessive 
in  relation  to  the  \olume  of  business  and  sell¬ 
ing  price  than  can  be  anticipated? 

6.  One  for  which  the  company  possesses 
the  manpower  and  facilities  to  develop,  manu¬ 
facture,  and  introduce? 

7.  One  for  which  the  development  and  in¬ 
troduction  costs  can  be  amortized  in  at  least 
a  normal  payout  period? 

8.  One  upon  which  the  return  on  investment 
can  be  estimated  with  reasonable  accuracy? 

9.  One  which  the  company  is  financially  in 
a  position  to  develop  and  introduce? 

Along  similar  lines  in  regarej  to  the  consideration 

of  the  development  of  a  new  product,  another  author  list 
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the  following  questions  to  be  answered: 

1.  What  reasons  are  there  for  moving  into 
the  new  type  of  business? 

2.  What  is  the  relative  status  of  present 
products? 

3.  Which  new  markets  are  most  closely  re¬ 
lated  to  those  now  being  reached? 

4.  What  is  their  expected  rate  of  growth? 

5.  What  is  the  status  of  competition  in 
the  new  business? 

6.  Are  there  needs  that  are  not  being  met 
by  available  products? 

7.  What  are  the  capital  requirements  for 
getting  established  ? 
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8.  IVhat  are  the  requires.ents  for  success 
in  the  new  business? 

9.  Are  we  properly  staffed  in  kind  and 
number  to  manage  a  successful  new  business? 

10.  Should  expansion  take  place  through 
acquisition  or  by  developing  our  own  products? 


One  research  consultant  suggests  the  use  of  a  check¬ 
list  approach  by  which  each  proposal  should  be  subjected 
to  rigid  tests  to  determine  technical  feasibility  from 
an  engineering  and  manufacturing  veiwpoint,  its  market¬ 
ability  based  upon  customer  tastes  and  distributive 
requirements,  and  its  profit  potential  based  on  projected 
sales  volume,  price,  cost  of  production,  and  cost  and 
availability  of  capital.  In  his  checklist  he  includes 
many  different  factors  and  lists  the  probable  source  of 
information  on  each  of  these  factors.  The  major  headings 
in  the  list  of  factors  include:  research  proper,  pro¬ 
duction,  customer's  specifications,  potential  demand, 
scale  of  plant,  location  of  plant,  operations,  competi¬ 
tion,  pricing  to  achieve  objectives,  capital  require¬ 
ments  and  profits,  relation  to  lojig-term  plan,  effect  of 
change  on  process  research,  measuring  gains,  financing 
capital  requirements,  and  risk.  The  major  sources  of 
information  for  these  factors  include:  research,  engi¬ 
neering,  market  research,  economic  research,  manufac¬ 
turing  and  sales,  new  product  development,  traffic,  ac- 
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counting,  budgeting,  and  treasurer. 


James  R.  Bright  goes  into  considerable  detail  in 
his  relatively  recent  book  on  technological  innovation  in 
discussing  the  appraisal  of  the  potential  significance 
and  possible  risks  in  technological  innovation.  .He 
presents  detailed  listings  of  various  considerations  in 


# 
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each  of  rheso  arcais.  One  listing,  relating  to  conditions 
for  achieving  economic  value  from  potential  technological 
advances,  includes  under  areas  of  consideration  the  effects 
on  product  performance,  effect  on  the  inputs  required  to 
produce  or  support  the  innovation,  effect  on  users,  and 
effect  on  economic  society.  Another  listing  relating  to 
risks  in  innovation  includes  checklists  of  risks  in 
technical  areas,  including  scientific,  engineering,  and 
production  risks,  risks  in  marketing  areas,  interference 
risks,  timing  risks,  and  obsolescence  risks. 

In  general,  then,  the  criteria  for  pre*appraisal  of 
contemplated  research  programs  include  a  measure  of  eco¬ 
nomic  potential,  taking  into  account  sales  and  profit 
potential,  return  on  investment,  and  intensity  of  compe¬ 
tition;  along  with  a  measure  of  marketability  or  customer 
acceptance  and  a  measure  of  technical  feasibility,  taking 
into  account  technical  obstacles,  cost  of  research,  time 
span  required  for  research,  and  the  possibility  of  obso¬ 
lescence  brought  about  by  the  introduction  of  a  competitive 
49  * 

product. 

Despite  the  previously  discussed  limitations  of  the 
use  of  quantitative  methods  in  the  pre-appraisal  of  re¬ 
search  projects,  many  companies  use  various  formulas  to 
either  determine  or  to  assist  in  determining  the  potential 
value  of  the  proposed  program.  One  of  the  better  known 

equations  is  the  "Index  of  Return  Formula"  devised  by  Fred 
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Olsen  of  Olin  Industries. 

EVR  X  ECS 


IR 


ECR 


where: 

IR  ■  Index  of  Return;  which  deterninos 

whether  a  project  is  worthwhile  or 
should  be  abandoned. 

EVR  •  Estimated  Value  of  Research,  if  it 

succeeds;  the  sutimaticn  of  the  full 
value  of  savin;;s  for  one  year  for  a 
process  improvement  plus  2  per  cent 
of  the  sales  value  for  two  years  of 
a  product  improvement  plus  5  per  cent 
of  the  sales  value  for  five  years  for 
a  new  product. 

ECS  ■  Estimated  Chance  (or  probability)  of 
Success. 

ECR  ■  Estimated  Cost  of  Research. 

Experience  has  shown  that  the  index  of  return  should 
exceed  the  value  of  3  before  the  project  is  consid¬ 
ered  worthwhile  and  the  project  should  be  abandoned 
should  the  index  fall  below  3.  The  percentages  and 
time  periods  for  the  EVR  term  are  arbitrarily 
assigned  and  have  been  changed  in  the  years  since  the 
formula  was  first  applied. 


A  variation  of  Olin's  Index  is  the  Miller  Variation  sug¬ 
gested  by  Theodore  Miller,  which  uses  as  a  numerator  the 
product  of  the  following: 

(Percentage  chance  of  technical  success)  X 
(Percentage  chance  of  commercial  success)  X 
(Estimated  unit  sales  per  year)  X 
(Profit  per  unit)  X 

(The  square  root  of  the  life  of  the  product 
stated  in  years) 

The  formula  yields  a  "product  number"  which  is  used 
as  a  basis  for  comparing  various  research  proposals. 


A  formula  described  by  Ralph  Manley  of  General  Mills 
is  used  as  a  guide  to  the  amount  of  money  that  a  company 
can  afford  to  spend  on  research  leading  to  a  new  develop¬ 


ment: 


.52 


R 


2  (0.01  YXS  -  P  -  h) 


where: 

R  «  research  and  development  costs  before 
tuxes , 

P  ■  plant  investment, 

h’  >  working  capital  required, 

Y  ■  recoupment  period  (period  in  which 
costs  must  be  recoveredj , 

S  •  annual  sales  volume  of  new  product,  and 
N  ■  minimum  acceptable  net  profit  on  sales. 

Using  representative  figures  provided  by  General 
Mills,  a  $1  million  plant  (P)  produces  $1  million 
worth  of  goods  per  year  (S) ;  to  run  such  a  plant, 
$0.5  million  working  capital  (K)  is  required;  the 
payout  period  (Y)  is  5  years;  the  minimum  return 
on  sales  (.\)  is  6  per  cent;  50  per  cent  of  research 
and  development  costs  ai'e  tax-deductible  (the  factor 
2  in  the  equation).  The  computed  amount  that  can  be 
spent  on  research  (R)  is  $600,000. 

G.  K.  Teal,  Assistant  Vice-President  of  Research  and 

Engineering  for  Texas  Instruments^  described  an  Index  of 

Research  Effectiveness  used  to  quantitatively  evaluate 

research  projects.  The  index  is  the  product  of  four 
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Other  indices: 

Index  of  Return  on  R  6  0  »  .N  /  25  S 

where: 

.\  ■  Xot  profit  during  the  life  of  the  re¬ 

sulting  product. 

S  ■  R  ^  D  costs. 

25  -  Taken  from  statistics  on  return  on  R6D 

collected  by  the  National  Science 
Foundation. 

Index  of  Return  on  Assets  ■  N  /  13. 5 t  A 
where: 

N  ■  Not  profit  during  the  life  of  the  re¬ 
sulting  product. 

A  ■  Assets  which  were  required. 

13.5^  -  Based  upon  a  judgment  that  a  worthwhile 
product  should  make  possible  a  favor¬ 
able  return  of  13. 5». 


Index  of  Dollar  Volume  =  b  /  (ii/25j 


where: 

b 

B 

25 


Billings  laade  possible  by  the 
product. 

Total  billings  during  the  product's 
li fetine . 


Based  upon  a  judg.T.cnt  that 
while  product  should  riaho 
a  4'd  increase  in  total  an 


a  v.orth 
poss iblc 
nua  1.  oil' 


lings . 


Index  of  Market  Capture  =  b  /  (M/2J 

where: 


b 

M 

2 


p  rouuc  t . 


Billings  r.ado  possible  by  the 
Total  availab.^e  !*4«* I* KO  w  • 

Based  upon  the  judg.7.ent  that  a  .orth- 
while  product  should  capture  1/2  of 
the  total  ziarket. 


If,  when  multiplying  these  four  indices  to  obtain  an 
index  of  research  effectiveness,  a  factor  of  one  or 
more  is  obtained,  the  project  is  considereu  a  worth* 
while  investment. 

The  Calculated  Risk  Basis  Formula  is  used  to  compute 
a  project's  rate  of  return  to  be  measured  against  a 
standard  which  states  an  acceptable  return 

n 

I  [?  (1^)] 
i  -  0 

R  •  - ^ - 

n 

I  (op 

i  -  0 

where: 

■  Net  estimated  income  from  the  project's 
^  results  in  any  _^th  year, 

n  ■  The  last  year  in  wnich  incomes  are 
expected, 

P  ■  Probability  of  receiving  the  ^th  income. 


# 


o 

o 
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0-  =  The  net  incremental  investment  out¬ 
lay  in  any  ^th  year, 

i  =  The  year  hence  in  which  the  income 
or  outlay  will  occur,  and 
R  =  The  project's  rate  of  return  on  total 
funds  invested. 


The  Unadjusted  Return  Techni 
lations  of  a  project's  potential 
Projects  are  more  desirable  as  R 

R  required  to  accept  any  project 

.  .55 

taxes : 


que  makes  broad  calcu- 
return  versus  its  costs, 
increases,  the  minimum 
being  30  per  cent  before 


n 


i  =  0 


n 

2  (0-) 
i  =  0 

where : 

R  =  The  rate  of  return  on  investments  in 
the  project, 

I.  =  The  inco.me  the  project  will  yield  in 
each  i^th  year, 

0-  =  The  incremental  'outlay  required  in 
each  ith  year, 

i  =  The  numFer  of  years  hence  each  income 
or  outlay  is  realiced,  and 

n  =  The  number  of  years  hence  the  last  in¬ 
come  or  outlay  is  realized. 


The  Hoskold  Transformation  computes  the  present  worth 
of  the  incomes  the  project  will  yield  for  comparison  with 
the  present  value  of  the  project's  actual  and  projected 
costs  to  determine  if  it  is  financially  acceptable : 


P 


D 


R' 


R  + 


(1  + 


where: 

P  ■  The  present  worth  of  t  '.c  incon.es  t;.e 
project  will  yield  if  successful, 

D  ■  The  average  an:*ual  incremental  inco.v.e 
yielded  if  the  project  is  successful, 
R*  «  The  average  net  return  on  capital  in¬ 
vested  in  the  enterprise, 

R  ■  The  current  rate  of  interest  on  invest¬ 
ments,  and 

n  ■  The  number  of  years  within  which  re¬ 
search  costs  must  be  recovered. 


Present  Value  Techniques  are  often  used  since  the 
(1  R)~^  factor,  shown  below,  is  readily  available  in 

standard  financial  tables  and  projects  can  be  ranked  in 
order  of  ascending  P  to  determine  their  relative  desira- 

n 

P  -  I  ICj  (1  ♦  R)'^  ] 
i  •  0 

where: 

P  ■  The  project’s  present  value, 

Cj  •  The  net  cash  inflow  in  each  ^th  year 
^  (net  cash  outflow  in  a  year  would 

make  Ci  negative), 

i  ■  The  number  of  years  hence  the  cash 
flow  occurs, 

n  ■  The  number  of  years  hence  the  last 
significant  cash  flow  occurs,  and 

R  ■  A  discount  rate  which  measures  the 

rate  at  which  the  company  woulu  in¬ 
vest  its  money  to  obtain  cash  flows 
of  equal  risk  in  the  same  year. 

A  framework  for  evaluation  of  applied  research  has 
been  proposed  by  I.  H.  Ansoff,  which  establishes  two 
formulas  based  on  criteria  related  to  potential  earr.i:;gs 
and  criteria  related  to  investment.  3y  assigning  arbi¬ 
trary  scales  to  the  individual  criteria,  both  figures  of 
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profit  merit  and  of  risk  may  be  computed  for  each  project 

CO 

and  then  the  competing  projects  may  be  compared. 


(Mt  ♦  X  E  X  Pg  X  Pp 

Cd  X  J 


X  S 


where: 


I. 


II. 

III. 


IV. 


Criteria  Related  to  Potential  Earnings 
Profit  potential 

Estimate  of  total  sales  (earnings) 
over  lifetime  (E) 

Technological  merit  (M^) 

Business  merit  (*M.  ) 

Probability  of  success  of  project  (P  ) 
Probability  of  successful  market  penetra¬ 
tion  (Pp) 

Strategic  fit  of  proposed  project  with 
other  projects »  products*  and  markets 
of  the  company  (S) 


I. 


II. 


Criteria  Related  to  Investment 

Direct  investment  (C,^  ♦  C^^  C^.  ♦  C„^ 

ar  pr  pa  pe 


♦  KC  ♦  F  •  C^) 


hC 

r 


Total  cost  of  applied  research 
effort 

Total  cost  of  product  research 
effort  (exclusive  of  applied 
research) 

Total  cost  of  product  development 
effort 

Total  cost  of  product  engineering 
effort 

Working  capital  required 

Total  cost  of  extra  facilities 
required,  such  as  staff,  build¬ 


ings,  etc. 

Joint  cost  effect  (J> 


J  ■  Savings  factor  in  direct  invest¬ 
ment  resulting  from  use  or 
sharing  of  existing  facilities 
and  capabilities. 


Despite  the  laany  quantitative  forniulas  used  in 
evaluating  research  projects*  jtost  coDiipanies  scesi  to  use 
them  as  aids  in  making  zotq  subjective  judgments.  Hugh 
F.  Colvin,  Executive  Vice-President  of  Consolidated  Engi¬ 
neering  Corporation,  pointed  out  that  an  important 
consideration  is  the  standpoint  of  time.  Although  tech¬ 
nical,  manpower,  facilities,  capital,  and  estimated  rates 
of  expenditure  considerations  are  important,  the  project 
that  may  be  sound  at  one  point  in  time  may  be  quite  in¬ 
appropriate,  for  various  reasons,  at  an  earlier  or  later 

CO 

date.  And,  Thomas  Moranian,  in  his  book.  The  Engi¬ 
neer  as  Manager,  emphasized  that  although  a  company  may 
be  in  business  to  make  a  profit,  return  on  investment  is 
not  always  the  governing  factor  in  the  selection  process. 
At  times  a  potential  new  product  that  does  not  have  a  com¬ 
parable  return  on  investment,  but  fulfills  the  .need  to 
broaden  a  company's  line  of  products,  may  be  selected 
because  it  helps  to  maintain  and  promote  the  general  sales 
level,  or  a  project  improving  the  quality  of  an  existing 
product  may  be  selected  because  it  enhances  a  company's 
position  or  prestige. 

• 

In-Process  Mcasurenent 

The  measurement  of  effectiveness  of  research  efforts 
on  projects  or  products  in  the  in-process  stage  of  the 
developmental  cycle  appears  to  be  much  more  difficult  than 
either  pre-appraisals  or  post-evaluations  of  research 
endeavor.  Most  means  of  measurement  arc  actually  instru¬ 
ments  of  control.  Quantitative  devices  are  essentially 
of  the  scheduling  or  budgeting  type^ and  qualitative  de¬ 
vices  include  all  sorts  of  program  reviews  and  progress 
reports . 


Although  quantitative  scheduling  devices  can  rarely 
be  utilised  to  any  great  degree  of  success  with  the  more 
fundamental  or  exploratory  forms  of  research,  much  of 
the  applied  research  and  developmental  effort  may  be  con¬ 
trolled  and  measured  by  the  use  of  Gantt  charts,  project 
milestone  schedules,  life-cycle  analyses,  or  computer¬ 
reporting  techniques. 

Gantt  charts  are  often  used  to  control  and  measure 
the  progress  of  smaller- sized  research  projects  or  are 
used  in  conjunction  with  computer-network  systems  in 
larger  projects  for  more  detailed  analysis  of  actual 
versus  scheduled  performance  within  smaller  segments  of 
the  total  network. Project  milestone  schedules  are 
similar  to  the  Gantt  charts  in  that  they  show  scheduled 
dates  for  completion  of  certain  activities  along  with  the 
percentage  of  completion  to  date. 

The  life-cycle  method  of  control  and  measurement  may 
be  instituted  for  larger  projects  when  sufficient  data 
have  been  accumulated  to  determine  the  pattern  of  the 
initial  cycle  of  the  project  development.  This  method 
gives  a  mathematical  model  of  the  overall  manpower  pat¬ 
tern  of  a  project  from  the  planning  and  specification 
cycle  through  the  design,  prototype,  and  release  cycles 
of  the  life  of  the  project.  It  is  based  upon  an  exten¬ 
sive  study  of  R^O  programs  which  found  that  the  ratios  of 
total  manpower  between  two  successive  cycles  are  very 
stable  for  a  particular  type  of  operation.  Once  the 
pattern  of  the  first  cycle  has  been  determined  through 
means  of  computerized  accounting  procedures,  the  pattern 

for  all  tho  subsequent  cycles  may  be  determined  by  the 
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following  mathematical  equations: 


y 

■ 

k  (1  - 

y' 

m 

2k  a  t 

where: 

y 

m 

Manpower  to  date,  in  man-months. 

y' 

m 

Manpower  for  each  month. 

k 

m 

Total  effort  for  the  cycle  in  man- 
months  . 

a 

a 

Parameter  governing  the  shape  of  the 
curve . 

t 

a 

Time  from  start  of  the  cycle  (in 
months) . 

e 

A  constant  (base  of  natural  loga¬ 
rithms  ■  2 . 78128 . . . ) . 

Whenever  the  size  or  complexity  of  a  program  justi¬ 
fies  their  use,  computerized  scheduling  and  reporting 
techniques  such  as  PERT,  CPM,  or  RAMPS  may  be  used  for 
analyzing  in-process  research  and  development  activities. 

The  PERT  (Program  Evaluation  and  Review  Technique) 

system  is  used  when  a  project  requires  a  network  involving 

hundreds  to  thousands  of  events,  and  it  is  also  sometimes 

used  for  planning  smaller  projects  having  only  a  very  few 

« 

events.  Although  the  PERT  system  may  be  used  without  em¬ 
ploying  electronic  computers,  its  effectiveness  is  facili¬ 
tated  by  their  use.  The  PERT/TIME  system  is  based  upon 
the  preparation  of  a  network  of  specific  milestone  events 
connected  by  necessary  activities  leading  from  one  event 
to  another.  The  timing  of  the  activities  is  normally 
based  on  a  weighted  value  of  three  time  estimates--the 
optimistic,  the  most  likely,  and  the  pessimistic- -made  by 
the  scientist  or  engineer  responsible  for  the  activity. 

The  longest  path  in  the  network  going  from  one  particular 
event  to  another  is  called  the  critical  path,  while  the 
other  alternate  paths  are  referred  to  as  the  slack  paths. 


♦ 


n 

\j 


SiioulJ  tiic 


estimated  completion  date 


resulting  from  the 


time  span  along  the  critical  path  be  later  than  the  re¬ 
quired  target  date,  resources  may  be  transferred  from 
the  slack  paths  to  the  critical  path  or  the  network  may 
be  redesigned  to  eliminate  certain  tasks  or  to  parallel 
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various  activities  in  order  to  correct  the  situation. 


The  pERT/COST  system  translates  each  activity  in 
the  PERT/TIME  network  into  manpow'er  hours  and  related 
costs.  The  total  network  time  can  be  translated  into 
dollar  figures  for  each  activity  and  a  budget  may  be 
developed  for  the  entire  project.  The  PERT/COST  system 
permits  the  various  alternatives  to  be  considered  in 

light  of  the  costs  involved,  as  well  as  the  schedule 
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considerations . 

Like  the  PERT/COST  system,  the  CPM  (Critical  Path 
Method)  takes  costs  into  account.  This  method  includes 
time  and  cost  rates  for  concluding  a  project--a  normal 
rate  and  a  crash  rate- -which  permit  comparison  of  cost 
against  varying  durations  of  the  jiroject  in  order  to 
select  the  optimum  alternatives.^^  The  availability  of 
resources  is  taken  into  account  in  the  RAMPS  (Resources 
Allocation  and  Multi-Project  Scheduling)  system.  This 
facilitates  decisions  regarding  the  allocation  of  re¬ 
sources  whenever  there  are  alternatives  which  may  minimize 
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the  cost  of  completing  the  scheduled  program. 

These  quantified  control  systems  are  often  criti¬ 
cized  by  various  authorities  who  contend  that  too  great 
a  reliance  is  placed  upon  their  use.  Although  the 
systems  present  mathematical  answers  which  would  seem  to 
clearly  delineate  the  available  alternatives  and  direct 
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the  selection  of  the  optirau?:}  solution,  they  arc,  in  fact, 
based  on  qualitative  inputs  and  are  thus  no  better  than 
the  quality  of  the  inputs  thejnselves.  For  instance, 
with  the  PERT  system  subjective  judgements  must  first  be 
made  in  establishing  the  network  of  events  and  activities 
and  in  making  the  time  estimates  witnin  each  path. 

Critics  of  these  systems  feel  that  a  great  amount  of  time, 
effort,  and  cost  is  expended  in  quantifying  data  which 
are  based  on  little  more  than  qualified  judgment.  The 
systems  are  only  as  good  as  their  inputs  and  no  matter 
how  much  they  are  mathematically  manipulated  or  computer 
programmed,  if  the  initial  judgments  establishing  the 
data  for  the  system  were  poor,  then  there  is  nothing  but 
"garbage- in;  garbage-out."  Thus,  many  feel  these  complex 
quantitative  systems  are  of  questionable  value  in  con¬ 
trolling  and  measuring  research  and  development  activities. 
They  feel  that  the  uncertainties  involved  in  research  work 
where  there  is  a  diversity  of  operations  and  the  lack  of 
any  identifiable  product  unit  indicate  that  no  concrete 
measure  of  R^D  output  can  be  devised. 

For  these  reasons  many  believe  that  other  control 
devices  must  be  used  to  evaluate  research  effectiveness. 
"What  can  be  measured  at  present  with  a  reasonable  degree 
of  accuracy  is  the  amount  of  resources  devoted  to  research 
and  development.  .  .  .  Since  the  output  .  .  .  cannot  be 
measured  at  present,  the  only  way  the  volume  of  research 

7 

activity  can  be  described  is  in  terms  of  its  inputs." 

Some  believe  the  only  quantitative  system  that  can  be 
used  is  that  of  establishing  extensive  budgetary  and  ac¬ 
counting  systems  which  monitor  the  research  expenditures, 
compare  them  to  pre-established  budgets,  and  report 


♦ 
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variances  between  actual  and  budgeted  rates.  In  the 
Handbook  of  Industrial  Research  Management,  Walter  B. 
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McFarland  stated: 

Through  the  budget,  management  can  control 
the  total  amount  of  money  spent  for  research 
and  it  can  direct  expenditures  to  make  sure 
that  funds  are  spent  in  the  way  management  wants 
them  to  be  spent.  This  is  accomplished  by  plan¬ 
ning  how  available  funds  are  to  be  used  and  by 
comparing  current  expenses  with  budgets  to  aid 
management  in  charge  of  research  to  keep  actual 
expenditures  within  limits. 

McFarland,  Manager  of  Research  for  the  National  Associ¬ 
ation  of  Accountants,  further  stated  that  the  purposes 

of  cost  control  in  its  application  to  research  activi- 
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ties  are: 

1.  To  make  sure  that  the  plan  expressed 
in  the  budget  is  followed  by  directing  funds 
into  projects  of  types  desired. 

2.  To  avoid  spending  research  funds  on 
nonproductive  or  nonresearch  activities. 

3.  To  stimulate  an  attitude  of  dollar- 
consciousness  so  that  research  personnel  will 
attempt  to  perform  as  much  research  as  possible 
for  the  funds  available.  ^ 

4.  To  keep  the  total  spent  for  research 
within  the  limit  set  by  the  appropriation  for 
the  period. 

Toward  this  end,  research  costs  are  classified  and 
accounts  arc  set  up  so  that  direct  reports  may  be  pre¬ 
pared  from  the  established  accounts.  Such  classification 

71 

is  designed  to  answer  the  following  questions: 

1.  How  much  was  spent  for  research  and 
development? 

2.  Who  spent  it?  (i.e.,  classification 
by  responsibility  for  cost  control). 

3.  For  what  were  the  costs  incurred?  (i.e., 
classification  by  nature  of  response). 

4.  How  was  the  effort  of  the  research  and 
development  organization  applied?  (i.e.,  class¬ 
ification  by  project  and  division). 


Once  the  budget  has  been  established  and  accounts 
and  classifications  of  costs  assigned,  data-processin;^ 
operations  ir.ay  be  inaugurated  in  order  to  prepare  tiir.cl 
reports  of  variances  between  actual  and  budgeted  expen¬ 
ditures  to  guide  iiianagewent  decisions.  for  instance, 
cost  control  techniques  may  be  used  in  conjunction  with 
scheduling  and  milestone  devices  to  facilitate  the  sole 
tion  of  possible  alternatives.  If  the  sc);eduling  syste 
indicates  a  serious  delay  that  must  be  resolved  by  al¬ 
ternative  means,  the  cost  control  data  can  help  provide 
an  evaluation  of  the  expected  cost  of  avoiding  or 
reducing  the  impact  of  the  delay  compared  to  the  cost  o 
the  delay  itself.  Raymond  Villers  cites  two  formulas 

that  may  be  used  for  time  and  money  variance  analysis  i 
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this  situation: 


After  the  optimum  alternative  has  been 
selected  two  formulas  can  be  used  to  summarize 
the  situation  and  to  show  clearly  iiow  the  tech 
nical  progress  is  related  to  actual  expendi¬ 
tures  of  time  and  money. 


The  formula  for  the  tim6  factor  is: 


where : 


V. 


°t  -  -  Ep 

the  predicted  tiiiie  variance 
scheduled  time  for  the  project 
time  already  spent  on  the  project 
amount  of  time  that,  on  the  basis 
of  the  revised  schedule,  will 
be  needed  to  complete  the 
pro j  ect . 


By  adding  V-^  to  the  expected  completion  date, 
one  can  show  the  revised  estimated  completion 
date.  This  revised  estimate  must  be  compared  to 
the  company's  commitments  and  to  the  needs  of  the 
market  before  it  can  be  definitely  considered  as 
acceptable . 


The  second  rorr.;ula  translates  the  first  one 
into  dollars. 

Vfa  *  °b  - 

where: 

V.  =  the  predicted  variance  from  the 

bud^^e  t 

0,  =  the  original  budget  for  the  period 

aP  =  the  actual  expenditures  since  the 

start  or  the  project 
=  the  estimated  cost  of 

The  variance  Vt)  may  be  due  to  the  fact  that 
At  exceeds  the  original  schedule.  It  may  also  be 
due  to  the  fact  that  the  research  people  have  re¬ 
ported  that  lit  to  be  revised.  In  any  case, 
the  variance  Vb  provides  the  information  needed 
and  will  be  included  in  the  reports  issued  at  the 
various  levels  of  management. 

While  the  budgetary  systems  may  be  used  to  help  man- 

angement  evaluate  possible  alternatives,  most  authorities 

are  quick  to  point  out  the  limitations  of  such  devices 

in  the  measurement  and  control  of  research  and  develop- 
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ment  activities.  James  B.  Quinn  has  stated: 

I 

The  research  budget  is  more  useful  ..s  a 
planning  device  than  for  control  purposes.  Except 
in  the  most  applied  phases  of  research,  estimating 
simply  cannot  be  performed  with  sufficient  accu¬ 
racy  to  make  the  research  budget  ...  a  suitable 
standard  against  which  to  determine  research  ef¬ 
ficiency  or  effectiveness.  .  .  . 

In  RqD  budgeting,  it  is  essential  to  recog¬ 
nize  certain  attributes  of  forecasts: 

1.  Effort  on  a  project  can  be  predicted,  but 
results  cannot  be. 

2.  The  research  budget  is  likely  to  be  accu¬ 
rate  in  the  aggregate,  but  not  in  detail. 

3.  Research  program  budgets  are  accurate  in 
the  short  run  only. 

4.  Forecasts  must  allow  for  program  flexi¬ 
bility. 


that: 


In  a  similar  vein,  IValter  h.  Xcrarlaad  cautioned 
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Recosnition  should  be  L;ivcn  to  the  iact 
that  the  budget  is  .  .  .  not  a  device  to 
measure  the  output  of  research  operations  .  . 
While  the  application  of  effort  can  be  con¬ 
trolled  through  the  control  of  expenditures, 
measurement  of  results  obtained  in  relation  to 
specific  expenditures  is  usually  uncertain 
and  often  impossible.  Certain  goals  in  the 
form  of  new  knowledge  may  underlie  the  alloca¬ 
tion  of  resources  when  the  budget  is  prepared, 
but  expenditures  .  .  .  are  not  contingent  upon 
realization  of  these  goals. 


Thus,  like  the  highly  quantitative  control  devices 
such  as  PERT,  CPM,  RAMPS,  and  others,  the  budgetary, 
accounting,  and  cost  control  devices  are  not  a  panacea 
in  themselves.  They  are  merely  reporting  techniques 
which  provide  management  with  quantitative  data  upon 
which  to  base  qualitative  judgments.  For  this  reason 
many  companies  heavily  rely  on  a  system  of  reports 
which  present  both  quantitative  and  qualitative  analyses 
of  the  status  of  in-process  research.  The  importance  of 
these  various  status  reports  is  emphasized  in  the  follow¬ 
ing  statement  by  Roy  H.  Walters,  Vice-President  of 
Research  and  Development  for  General  Foods  Corporation:  ^ 

These,  then,  arc  the  two  extremes:  (1)  the 
report  we  write  before  we  start  work  to  be  sure 
we  know  what  we  are  going  to  do;  and  (2)  the 
final  report  in  which  we  put  dow'n  what  we  think 
we  did.  The  one  is  too  early  to  be  of  major 
use  in  keeping  research  on  the  track,  the  other 
is  much  too  late.  It  is  the  great  space  between 
these  two,  the  so-called  during- the  -  fact  report, 
that  is  most  vital  to  research  manage:'.ient . 

Toward  this  end,  Harley  H.  Bixler,  Manager  of 
facturing  Process  Development  at  General  Electric, 


Manu- 
s tated : 


# 


There  are 
report  project 


several  way:i  to  evaluate  and 
s  tatus : 


1.  The  pro-;ress-ai;ai:;st-schedule  report, 

2.  A  periodic  reviev;  or  r.iaauTacturinii 
which  is  attended  by  laboratory  executives  as 
well  as  by  the  project  engineer, 

5.  A  pi'ogress  report  which  is  reviewed 
by  the  project  engineer  and  laboratory  execu¬ 
tives, 

4.  A  continuing  review  by  lab  executives 
through  visits  to  the  project  section. 


In  general,  then,  despite  the  limitations  of  any 
particular  method  of  measuring  research  effectiveness, 
by  employing  a  system  of  financial,  technical,  and 
scheduling  reports  and  reviews,  management  can  keep 
abreast  of  the  progress  or  status  of  research  endeavor 
and  have  available  the  necessary  data  for  qualified 
judgment  and  informed  decison-making  in  measuring,  con¬ 
trolling,  and  directing  the  research  and  development 
function. 


Post- Evaluation  < 

The  post-evaluation  of  research  efforts  is  perhaps 
somewhat  more  concrete  an  evaluation  than  can  be  per¬ 
formed  at  the  pre-program  and  in-process  stages  of  the 
developmental  cycle.  In  theory,  the  research  has  led  to 
some  end  product  --  a  new  or  improved  product  or  process, 
or,  simply,  research  papers  which  provide  some  insight  or 
new  knowledge  --  which  may  be  evaluated  against  the 
actual  expenditures  incurred  in  conducting  the  research 
versus  the  original  estimate  of  expenditures  that  was 
made  when  the  proposed  research  was  initially  considered. 
Raymond  Villers  has  described  such  evaluation  as  follows: 
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In  ciaking  a  postappraisal,  the  several  esti¬ 
mates  of  return  on  investment  which  were  calcu¬ 
lated  in  the  first  analysis  of  the  project  ,a;.J 
in  subsequent  analyses  when  progress  was  reviewed 
and  additional  ivijD  funds  were  requested}  are  com¬ 
pared  with  the  several  returns  on  investment 
which  were  achieved  by  using  several  fi^^ures  for 
the  investment  in  research.  One  such  figure  will 
be  the  total  cost  of  research  from  the  time  tiic 
project  was  product  oriented  while  other  invest¬ 
ment  figures  will  include  the  cost  of  researci; 
since  the  various  appraisals  wore  made.  The  re¬ 
turn  on  investment  figure  which  is  based  on  the 
total  cost  of  research  since  the  project  was 
product  oriented,  even  though  some  of  the  R^D  was 
nonprogrammatic ,  is  useful  for  management  when 
it  is  combined  with  other  postappraisals  to  de¬ 
termine  the  overall  return  from  product-oriented 
research  and  to  measure  the  benefits  resulting 
from  R§D  activities. 


In  actuality,  however,  such  quantitative  post-evaluation 
of  individual  projects,  described  by  Villers,  arc  beset 
with  difficulties.  Although  quantitative  measurements 
of  effectiveness  may  be  used  vrith  some  degree  of  success 
in  evaluating  the  applied  and  developmental  types  of 

i 

research,  they  are  of  little  value  in  analyzing  those 
projects  falling  in  the  more  fundamental  researci;  end  of 
the  spectrum  of  uncertainty.  In  discussing  the  limita¬ 
tions  of  quantitative  analysis  methods,  James  3.  Quinn 
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Stated: 


Although  some  quantitative  devices  purport 
to  assess  past  fundamental- research  programs,  no 
company  has  found  a  satisfactory  way  to  measure 
the  impact  of  research  performed  less  than  three 
to  five  years  previously. 

Even  in  other  areas,  formulas  provide  answers 
based  on  figures  that  are  themselves  inaccurate. 
One  can  seldom  predict  precisely  how  much  a  new 
product  will  earn.  No  sound  method  exists  for 


upport ioair.i;  croJir  Cor  a  product's  profit  bc- 


tweor*  a  coir.paiiy's  research  department  and  its 
other  functional  ‘groups.  And  finally,  one 
cannot  allocate  research  costs  among  various 
products,  except  subjectively  and  imprecisely. 
However  impressively  mathematical  they  look, 
formulas  based  on  such  figures  provide  answers 
no  sounder  and  no  more  trustworthy  than  the 
personal  judgments  that  underlie  their  figures. 


Walter  B.  McFarland,  in  discussing  research  cost 
accounting  and  control,  listed  several  problems  in  match¬ 
ing  research  costs  with  sales  income,  among  them  the 
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following: 

Benefits  from  specific  research  expendi¬ 
tures  are  always  uncertain  and  years  may  pass 
before  success  or  failure  of  a  project  becomes 
apparent.  .  .  .  Many  projects  which  fail  to 
accomplish  their  objectives  yield  incidental 
knowledge  which  proves  to  have  great  value  at 
some  future  time.  Moreover,  the  useful  life 
of  knowledge  gained  by  research  cannot  be  pre¬ 
dicted  with  sufficient  reliability  to  guide 
amortization  of  costs  over  the  future  periods 
benefitted. 


For  these  reasons,  many  organizations  make  little 
attempt  to  quantitatively  evaluate  individual  research 
projects,  but  try  instead  to  measure  the  output  of  the 
total  research  function.  Their  approach  is  to  view  re¬ 
search  success  in  light  of  overall  company  progress, 
without  arbitrary  establishment  of  payout  and  amortiza¬ 
tion  periods  and  allocation  of  research  and  overhead 
costs  for  a  particular  project  and  without  artificial 
apportionment  to  the  project  of  the  benefits  and  profits 
resulting  from  the  ultimate  production  and  marketing  of 
the  research  results.  Many  different  formulas  and  quan¬ 
titative  measuring  devices  are  used  by  various  companies 


and  organizations  to  evaluate  the  overall  ei fecti veness 
of  tho:r  research  eJforts.  Air.ong  then;  are  the  follow¬ 
ing.'^^ 

Comparison  of  research  expenditures  with 
sales  gain  over  a  period. 

Comparison  of  research  expenditures  wit); 
the  ratio  of  sales  of  new  products  to  the 
sales  of  all  products. 

Comparison  of  research  expe.ndi.tures  witf. 
the  change  in  market  share  over  a  given  t-!.me 
period. 

Comparison  of  research  expenditures  with 
selling,  advertising,  or  manufacturing  expendi¬ 
tures  . 

Comparison  of  the  rate  of  net  profit  from 
investment  in  research  with  the  current  rates 
of  return  on  plant,  property,  >..quipment,  or  in¬ 
vested  capital. 

Comparison  of  the  number  of  patents  with 
those  of  previous  periods  of  time  or  with  those 
of  competitors. 

Number  of  disclosures  to  the  patent  at¬ 
torney  by  the  research  personnel. 

Number  of  scientific  papers  presented  out¬ 
side  the  organization. 

Amount  of  royalties  received  from  other 
companies  from  licensing  of  research-developed 
products. 

Dollars  of  sales  from  new  products. 

Trend  of  new  product  sales  measured  as 
percentage  of  total  annual  sales. 

Increase  of  the  company’s  share  of  the 
market  for  specific  new  products. 

Rf7nking  of  the  company  within  its  indus¬ 
try  and/or  market. 

Comparison  of  the  company’s  research  posi¬ 
tion  with  that  of  other  companies  by  various 
research  intensity  ratios,  such  as  the  number  of 
people  engaged  in  researcl;  or  by  dollars  of  RdD 
expenditures  against  operations  in  terms  of  em¬ 
ployment  or  in  dollars. 


In  a  survey  of  37  laboratories  conducted  by  the  Mass 
achusetts  Institute  of  Technology,  A.  H.  Rubenstein 


reported  ti'.e  loliowiiii;  categories  of  criteria  used  to 


j  udi;e  research  aad  development  work; 


Number  of 
Companies 

(1)  Aclated  to  effect  on  sales  vo I ume 

or  revenue  .  19 

Increased  business;  increased 
output  without  increasing  investment; 
share  of  the  market;  percent  of  prod¬ 
ucts  from  research;  consumer  accep¬ 
tance;  effect  of  ne\c  products  on  old 
product  sales;  new  customers. 

(2)  Re lated  to  c f feet  on  savings  in 

materials ,  labor,  or  other  costs .  17 

Royalty  payments  saved;  use  of 
by-products,  wastes,  idle  facilities 
or  personnel,  or  less  profitably  em¬ 
ployed  facilities;  reduction  of  prod¬ 
uct  line;  closer  control  of  manufac¬ 
turing  quality;  better  process  yields, 
etc. 

(33  Related  effect  on  profits .  13 

Profit  on  research  versus  non¬ 
research  products;  profit  and  loss 
analysis  for  whole  R^D  effort;  payoff 
time  on  projects;  perce*nt  return  on 
investment . 

(4)  Related  to  time  and  cost  of  the  tech¬ 
nical  solution .  28 

Frequent  re-estimates  of  time  and 
cost;  progress  on  project  or  program 
phases;  actual  versus  budgeted  ex¬ 
penses;  actual  versus  scheduled  prog¬ 
ress;  proportion  of  budget  spent  ver¬ 
sus  progress. 

(5)  Related  to  customer  satis  faction .  10 

.Number  and  nature  of  complaints; 
broadening  of  product  line. 

(6)  Related  to  information  output .  17 

.Number  of  valuable  ideas;  percent 
of  ideas  from  inside  laboratory;  learn¬ 
ing  about  new  processes  and  materials; 


sources  of  nev  ideas;  traiainj;  indi¬ 
viduals;  devclopn.ent  of  spcci  fications  ; 
evaluation  of  inforr.iation  outnut  to 
application  groups;  inforina ticn  devel¬ 
oped  for  sales;  repeat  requests  for 
work. 

(7)  Related  to  s u c c e s s  of  techni. cal 

operatfons  . .  lb 

Number  of  problems  successfully 
handled;  number  of  patents  - -wri tten 
up,  applied  for,  granted;  number  and 
nature  of  project  failures. 

Less  quantitative  evaluations  are  employed  by  many 

organizations.  The  American  Institute  of  .Management  uses 

no  formulas  in  analyzing  research  success;  instead,  it 

attempts  to  determine  those  aspects  of  company  progress 

which  may  be  properly  credited  to  research  by  asking  the 
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following  questions: 

1.  1/hat  technological  innovations  has  the 
company  made  that  are  now  used  in  its  industry? 

What  basic  discoveries? 

2.  What  equipment  or  methods  has  it  devel¬ 

oped  for  its  own  operations  as  a  result  of  re¬ 
search?  ' 

3.  What  patents  have  been  granted?  liow  many 
each  year? 

4.  What  proportion  of  present  revenues  de¬ 
rive  from  products  developed  by  research?  What 
proportion  of  the  company's  products  derive  from 
its  research? 

5.  What  change  in  market  position  can  be 
attributed  to  research? 

A  broader  qualitative  approach  is  that  taken  by  £. 
Duer  Reeves,  Executive  Vice-President  of  Esso  Research  an 
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Engineering  Company,  in  the  following  two  statements: 

.  .  .  modern  industrial  research  is  only  a 
part  of  an  integrated  company  effort.  Trying  to 
measure  its  effectiveness  by  how  much  money  the 


1. ; 
o  o' 


coiripany  makes  or  by  how  m.any  now  products  are 
put  on  the  market  is  like  ^jivini^  a  football 
quarterback  all  the  credit  every  time  his  team 
scores  a  touchdown. 

There  are  five  questions  I  think  pretty 
well  define  the  job  of  the  research  or^ianization 
and  I  think  the  answers  to  them  :4ivo  a  pretty 
good  idea  of  just  how'  effective  it  is: 

1.  Does  the  research  organization  know  what 
technology  its  company  needs? 

2.  Is  the  research  organization  creating 
the  needed  technology? 

3.  Is  the  research  organization  helping  to 
get  research  results  applied? 

4.  Are  the  research  organization's  opera¬ 
tions  efficient? 

5.  Is  the  research  management  an  effective 
part  of  the  executive  team? 

Perhaps  the  broadest  qualitative  approach  toward 
measuring  the  effectiveness  of  the  research  organization 
is  presented  by  h'arren  C.  Lothrop,  in  his  book,  Management 
Uses  of  Research  and  Development : 


I  believe  that  the  R^D  organization  must  be 
judged  in  its  own  environment .  This  can  be  con¬ 
sidered  on  three  levels:  the  national  or  cultural 
environment,  the  industry  environment,  and  the 
company  environment.  .  .  .  V.’e  should  not  judge  the 
worth  of  these  three  environ:r.ents ,  or  societies, 
in  considering  the  function  of  research  and  devcl- 


vaiue 


the  function  in 


concrete  accoiznlis.'.z'.cnt  tna' 


serves 

tl . 
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is 


opment;  rather  we  should 
terms  of 

needs  of  that  particular  society.  By  "society 
mean  any  group  that  shares  a  com;r.on  purpose.  Such 
groups  can  be  national  groups,  ir.dustry  grouns 
company  groups.  If  the  technical 
a  part  of  one  of  these  groups,  its  purpose  must  be 
in  accord  with  the  purpose  of  the  group,  and  its 
accompl ish:'.icnts  must  be  judged  in  terms  of  its  ben¬ 
efits  for  those  who  compose  the  group.  In  other 
words,  it  must  advance  the  needs  of  its  own  society 
otherwise  it  has  no  validity  or  real  purpose.  lival 
uation  of  its  success  will  naturally  be  in  terms  of 


how  satisfactorily  it  meets  these  social 


c  C  A 

needs . 


Individual  Croativitv 


Uhcrcas  the  i7iCtnocIs  oi  ovaj-uat^on  a  is  cuss  eel  prc'c^G'^^s  * 
ly  are  means  of  mcasureraont  relating  :..ore  directly  to  the 
research  organization's  ohjcctivcs  and  res 
measurement  systems  are  related  to 


I  »  T 


L.  ^  f  a'I  V 


I  A  C  ^  i  c*  ^  L*  c* 


ideas  and  performance  of  individuals  ’aiti.in  ti.e  research 
organization.^^  Merit,  contribu  LxO«a>  ^yAOcAL4Cc^\-A.Ly^ 

rating  systems  are  often  used  to  measure  individual  re¬ 
search  ef forts. Other  m  0  ^  1  0  ^  ^  ^  ^  ^  ^  ^  ^  ^  ^  Ct  i  i  Lr  U~  CCa  C>-A.  \  W 
measurement  of  research  proposals  submitted,  papers  pub¬ 
lished,  patents  applied  for,  and  similar  measures  of 
individual  creativity  are  also  used. 

However,  the  value  of  these  particular  evaluation 
methods  is  being  increasingly  questioned  by  behavioral 
and  organizational  scientists,  C^riC*  WIaI^W 

use  may  even  be  detrimental  to  ti'.c  effectiveness  of  t;ic 
research  effort.  For  ins  c  ci  IaOO|  *  i  ^  iy  4300i\j  V-^I.  ^^c*^A  A  * 

and  Innovation,  Chris  Argyris  asks,  ".  .  .  what  is  the 
meaning  of  criteria  used  for  assessing  creativity  that 
depend  upon  superiors'  ratings,  papers  published,  and 
projects  finished?"^"  Few  researchers  believe  the  ratings 
by  either  supervisors  or  peers  have  any  validity,  citi.er 
because  of  the  pressures  on  the  rater  him.sclf  or  his  lack 
of  knowledge  and  understanding  of  t'ne  work  the  rcsearci.er 
is  performing.  Argyris  describes  a  sel  f intaining 
system  leading  to  greater  and  greater  ineffectiveness: 

The  imposition  of  research  evaluation  s>ste;..s  en,;e:;ders 
psychological  reactions  of  defense  and  dii>^rust  wi'.ici; 
lower  interpersonal  competence  and  researci;  effect! ve;. ess  , 
which  causes  management  to  impose  still  more  controls 
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which  create  deeper  defense  reactions  which  lower  research 

88 

effectiveness,  and  so  on,  infinitum. 

Upon  reviewing  recent  trends  of  increasing  pressure 

upon  research  organizations  to  become  more  efficient, 
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Argyris  feels  that: 

Increasing  intervention  by  the  line  depart¬ 
ments,  tighter  research  budgets,  more  systematic 
evaluating  research,  more  rigorous  (hopefully) 
quantitative  indices  for  evaluating  research  pay¬ 
off,  more  use  of  charts  to  control  the  flow  of 
research,  closer  links  with  the  marketplace  and 
a  weaker  tie  with  basic  research,  and  finally  an 
increase  in  the  use  of  top-level  committees  to 
oversee  research,  can  be  expected. 

I  question  if  these  activities  will  solve 
the  problems  of  research  organizations.  .  .  . 

As  budgets  tighten,  pressures  increase,  as¬ 
signments  to  work  become  (from  the  researcher's 
point  of  view)  more  random,  evaluative  procedures 
increase,  the  researcher  is  forced  either  to  give 
up  and  permit  himself  to  be  pushed  and  pulled  as 
the  pressures  go,  or  to  retreat  into  his  own  work 
and  insulate  himself  from  others.  .  .  . 

f 

It  seems  to  me  that  the  strategies  of  tight¬ 
er  controls  may  increase  the  control  that  manage¬ 
ment  has  over  already  (and  increasingly)  ineffi¬ 
cient  activities.  The  unintended  consequences  . 

.  .  will  develop  and  strengthen  the  undesirable 
and  uncontrollable  defensive  activities  that  can 
make  the  organization  less  flexible,  changeable, 
open,  and  innovative.  It  is  not  too  farfetched 
to  predict  that  if  these  processes  continue  long 
enough,  they  could  help  to  make  research  organi¬ 
zations  sick  organizations. 

Argyris  believes  that  the  imposition  of  necessary 

controls  is  not  the  problem;  rather  it  is  one  that  is  con- 
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cerned  with  the  climate  of  the  organization  itself: 


I  ar.  not 
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cut 


,  ^  C  ci  1 

ol3JOCtl\^OS)  ctllv-fc  •I-4-4--4-U  C\vi-*-k4u»w-l.o«*  ^yXOCc^^IOS 

not  useful.  They  arc  necessary,  ar.d  the  loss 
defensive  an  or'^anization  is,  tr.c  ;:.Gre  it  can 
generate  and  accept  its  Gv;n  CG:.trols.  I  a;;,  i:..- 
P 1  y  r  n  g  t  n  ci  w  *  c*  •*/  ^  o  r  c  c*  s  o  a  .  e  e  ^  .  e  e  ^ 
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be  aecu..cite,  ...uv^  re^e«-.*Ci. 

I3  e  r  a  ^  ^  a  ,  i. . .  a  V  a ..  .a**a.  .a^  a  ...  a.i^.  wa.ita. 

re  1  a 1 1 G ns i'i  1  p s  v.'i.,..  oe  i;.av.e  ei.ee ti'/e  ay  ta.a'.se 
corrective  rieasures.  Vhese  n.casure.s  do  not  get 
at  the  heart  of  the  prcblen..  At  the  core  of  the 
problem  is  the  basic  tendency  for  resa-archers  to 
be  uncomfortable  with  leeli  ciiiLA 

risk  taking  in  interpersonal  relationships. 

This  leads  to  a  research  culture  that  will  tend 
to  reward  conformity,  mistrust,  antagonism,  and 
fear  of  risk  taking.  As  fear,  mistrust,  and 
frustration  v;ith  others  increases,  the  indivi¬ 
duals  begin  to  question  the  motive  of  others. 


As  an  answer  to  this  problem,  Argyris  urges  that  organ! - 

ne 
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zations  must  e.xamine  themselves  and  cast  off  their  organi 


zational  defenses: 


Xo  W«iV' 

especially  the  research  a^:;.inis  t raters ,  be  i'.elp- 
ed  to  increase  their  interperson..l  competence 
and  decrease  the  incongruence  between  their 
values  and  their  behavior.  .  .  .  Hacii  organiza¬ 
tion  .  .  .  must  look  within,  at  precisely  tr.ose 
factors  that  it  has  been  u.iabl.'',  to  date,  to 
discuss  openly.  .\n  o;\.,ani zat: on  car.  ,.;o  from 
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reallv  feel,  becoir.e  m.ore  0  len  and  rece”.tive, 
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duality t  co  ncern,  and  tri.Sv. 

These  views  held  by  Argyris  were  extracted  in  ra 


great  detail  bec..use  they  pose 
the  measurement  and  control  0.' 
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and  techniques  that  :.iay  be  employed  to  appraise,  measure, 
and  evaluate  the  research  performed  within  an  organiza¬ 
tion.  The  implicit  assumption  has  been  that  research 
effectiveness  can  be  measured  by  some  sort  of  method  or 
device.  Argyris  raises  the  question  that  the  use  of  any 
method  or  technique  of  measurement  and  control  may  be 
counterproductive  if  the  organizational  climate  is  not 
conducive  to  the  use  of  such  devices.  Thus,  merely  by 
trying  to  measure  research  effectiveness,  the  effective¬ 
ness  itself  may  be  reduced  as  a  consequence. 


Those  who  are  concerned  with  the  evaluation  of  re¬ 
search  and  development  activities  might  take  comfort  in 
thinking  that  Chris  Argyris  is  alone  in  his  views.  Un¬ 
fortunately,  however,  many  others  share  his  opinions  and 
hold  similar  views  concerning  the  use  of  controls  (and 
associated  measurement  devices)  in  scientific,  research, 
and  highly  technically-oriented  organizations.  For 
instance,  Louis  S.  Barnes,  Division  of  Research,  Harvard 
Graduate  School  of  Business  Administration,  questions  the 

i 

usefulness  of  management  orientations  which  stress  profit 

productivity,  and  practicality  to  the  exclusion  of  other 

values.  In  light  of  the  negative  consequences  of  such 

actions  in  technical  groups,  Barnes  arrives  at  the  follow 
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ing  conclusions: 


Conditions  of 
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congruence 


_  in 

technical  groups  may.  offset  a  desired  output  of 
ideas,  imorove.7.cnts ,  and  jroduct  development. 

2.  Management  may  defeat  its  own  goals  by 
stressing  organizational  values  to  groups  whose 
backgrounds  encourage  their  seeking  other,  more 


uistant  reference  groups. 

5.  .Reward  systc.ms  in  relatively  closed 
systems  may  artificially  rc'..\.rd  an  individual 
for  his  social  status  rather  than  for  his  actual 
work  contributions. 


4. 


Or.^ar.i  zat  ior.al 


restrict  an  :.nciviaua 
subgroup  activity  ::;ay  ...c:.ev:.t 
tion  and 
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groups  Hiuy  iiitrouucc  supervisory  ucurroi  1  -4:..^ us 
at  the  expense  oi  saboru.^nate  aator.G;./.'  ar.v.  i^evei- 
opment . 
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procedures  run  the  risk  oi  ceniusin^  enus  v.ith 
efficient  inforr.'.ality . 
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departments  may  be  avoiuuu  by  structures  and 
procedures  v.hich  help  establish  departmental 
goals  rather  than  subgroup  goals. 
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productivity  .may  do  so  at  the  expense  of  a 
superior's  capacity  to  -L SiAUL/4.K.*rA4b*cOS  cdOuruvo 
that  productivity. 
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The  consensus  o  ^  4l4«ii«^  W4*Uw  Wi*0  Sw(.«44^CilC4 
and  evaluation  procedures  applicable  to  most  large  scale 
organizations  arc  inappropriate  for  research  efforts.  .In 
environment  co.mparatively  free  of  structural  rigidities 
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IS  more  conoucivc  to  scicntii^c  creativity. 

In  studying  barriers  to  innovation,  Sayic.-i  and  bobber 
found  that  the  innovator  is  often  censtraineu  by  organic.a- 
tional  structure,  tradition,  and  inertia,  and  that  cre..- 
tivity  in  the  industrial  environmer.t  i.s  4*^  4IIV4C«1  li  aU«4WW4044 
of  structural  design,  behavior..!  pat  c04«»Sy  V«i4W* 
pressures »  as  it  is  of  iudiviuuA*!  co::\]}cic\wc . 
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Creative  activity  cannot  be  forced.  It  requires  a 

favorable  climate  in  which  creative  people  --  the  free, 

unchanneled  minds,  the  nonconformists,  the  individualists, 

the  uninhibited  --  may  work  and  thrive.  The  frustrations, 

disagreements,  and  failures  which  often  result  from  a 

poor  climate  rarely  occur  in  an  atmosphere  which  is 

tolerant  of  "oddball"  individualists  and  is  receptive  to 

new  ideas,  which  is  relatively  free  of  the  pressures  to 

produce  and  permits  freedom  to  choose  problems  and  change 

the  direction  of  research,  which  recognizes  the  importance 

of  research  and  the  professional  status  of  the  researcher, 

and  which  rewards  creativity  and  enhances  mutual  respect 
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between  scientists  and  management. 


The  question  arises  as  to  how  research  .management  can 
foster  an  optimum  climate  for  the  development  of  creativi¬ 
ty  and,  at  the  same  time,  give  direction  and  speed  to  a 
research  group  which  performs  at  peak  effectiveness  in  the 
absence  of  rigid  controls  and  the  presence  of  freedom  of 
inquiry?^^  The  hallmark  of  an  effective  research  leader 
is  the  recognition  that  one  cannot  ".manage"  and  "direct" 
a  group  whose  idea  of  a  boss  and  of  o.xtcrnaI  controls  is 

an  anathema  inconsistent  with  the  basic  tenets  of  profes- 
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sionalism. 


It  is  only  in  c.mploying  a  leadership  style  which 
creates  an  at.mosphcre  of  mutual  respect,  integrity,  and 
confidence  that  the  manager  hope  to  provide  effective 
direction  to  the  research  effort,  he  must  create  a 
climate  of  "research-mindedness"  in  whicl.  the  needs,  op¬ 
portunities,  and  rewards  which  motivate  professional 
people  are  understood  and  apprec«.*ted. 


A  study  of  leadership  styles  in  research  crganiza* 
tions  bears  out  this  approach.  Participatory  leadership, 
characterized  by  a  high  degree  of  interaction  and  in¬ 
volvement  with  subordinates  and  joint  decision-making 
practices,  was  found  to  be  superior  to  either  laissez- 
faire  or  directive  leadership  patterns.  The  results  of 
the  study  confirmed  that  professional  personnel  do  respond 
to  situational  factors  in  organizations  and  that  the 

climate  resulting  from  participatory  leadership  styles 
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produces  favorable  motivations  and  attitudes. 

What  is  the  significance  of  these  findings  to  the 
establishment  of  methods  for  the  evaluation  of  research 
effectiveness?  Does  it  negate  their  use?  The  answer  is 
found  by  returning  to  the  statement  made  by  Chris  Argyris 
that  controls,  objectives,  and  evaluation  procedures  are 
useful  and  necessary,  and  that  the  less  defensive  an 
organization  is,  the  more  it  can  generate  and  accept  such 
controls.  Thus  once  an  optimum  climate  for  creativity 
and  innovation  is  established  by  an  understanding  and 
sympathetic  research  management,  then  appropriate  evalua¬ 
tion  techniques  may  be  used  without  the  danger  of  decreas¬ 
ing  the  very  effectiveness  they  attempt  to  measure. 
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IV.  Conclusion 


This  report  reveals  that  many  different  techniques 
are  employed  to  predict  and  determine  the  results  achieved 
from  scientific  endeavor.  Many  various  means,  ranging 
across  the  quantitative  to  qualitative  spectrum  of  method¬ 
ology,  are  used  to  appraise,  measure,  and  evaluate 
research  and  developmental  efforts.  It  is  important  to 
remember  that  no  one  method  is  a  panacea  --  each  has  its 
own  limitations  as  well  as  its  advantages.  There  is  no 
one  best  method  to  be  employed  in  analyzing  research  and 
scientific  endeavor.  The  particular  method  or  combina¬ 
tion  of  techniques  to  be  used  is  dictated  by  the  given 
situation.  Variables  which  affect  the  methods  selected 
for  analysis  incude  such  things  as  the  type  of  research 
agency  involved,  the  level  of  research  effort,  and  the 
stage  of  the  research  life-cycle  to  be  evaluated.  A 
method  which  might  be  quite  appropriate  under  one  set  of 
variables  may  be  completely  inapplicable  under  another 
set  of  conditions. 

Whether  more  quantitatively-oriented  methods  of 
analysis  will  be  employed  or  whether  more  qualitatively- 
determined  judgments  will  be  rendered  depends,  to  a  large 
degree,  upon  the  spectrum  of  uncertainty  --  both  in  rela¬ 
tion  to  the  levels  of  research  effort  and  the  stages  of 
the  developmental  cycle.  Therefore,  the  implications 
for  evaluation,  discussed  in  the  preceding  sections  of 
this  report,  will  be  again  summarized  so  that  the  impor¬ 
tant  interrelationships  between  these  variables  may  be 
more  readily  appreciated. 
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Levels  o i  Research  Kf fort 

It  has  been  pointed  out  that  the  degree  of  uncer¬ 
tainty  of  the  results  of  research  appears  to  be  the 
basic  criterion  for  categorizing  the  levels  of  research 
effort.  At  one  end  of  the  spectrum  of  uncertainty  is 
fundamental  research  and  at  the  other  end  is  develop¬ 
mental  application.  Assuming  that  the  spectrum  ranges 
from  a  great  deal  of  uncertainty  at  the  lower  end  to  a 
fairly  high  degree  of  certainty  at  the  upper  end,  then 
the  principal  implication  for  evaluation  of  research 
performance  lies  in  the  fact  that  research  results  may 
be  measured  with  relative  confidence  at  the  upper  end  of 
the  spectrum  whereas  the  measurement  of  research  effec¬ 
tiveness  becomes  progressively  more  difficult  toward  the 
lower  end  of  the  spectrum  of  uncertainty. 

Thus,  developmental  research, which  lies  at  the  upper 
end  of  the  spectrum  and  is  closely  concerned  with  produc¬ 
ing  a  product  whose  output  may  be  reasonably  well  compared 
to  the  inputs  required  to  produceiit,  can  be  effectively 
evaluated  by  employing  the  more  quantitative  methods  of 
analysis.  However,  fundamental  research,  which  falls  at 
the  lower  end  of  the  spectrum  and  which  is  concerned  with 
basic  inquiry  in  its  purest  sense,  must  be  evaluated  with 
the  more  subjective,  qualitative  methods  of  judgment.  The 
close  relationship  between  the  levels  of  research  effort, 
the  spectrum  of  uncertainty,  and  the  methods  of  evaluation 
employed  is  illustrated  in  Figure  1. 

It  was  also  pointed  out  that  research  efforts  are 
conducted  by  many  different  agencies- -  including  the 
government,  academic  institutions,  private  organizations. 
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and  industry.  The  principal  implication  concerning  the 
evaluation  of  research  effectiveness  is  essentially  the 
same  as  that  arising  from  the  spectrum  of  uncertainty. 

The  more  basic  the  research  performed  by  academic  insti’ 
tut ions,  some  government  laboratories,  and  a  few  indus* 
trial  enterprises,  the  more  subjective  or  qualitative 
must  be  the  means  of  evaluating  these  research  efforts. 

On  the  other  hand,  the  applied  research  and  development 
efforts  conducted  by  many  industrial  organizations  as  well 
as  by  some  of  the  other  types  of  agencies,  may  be  guaged 
by  the  more  quantitative  methods  of  analysis.  Because 
the  methods  employed  by  these  agencies  relates  more  to 
the  level  of  research  being  conducted  than  to  the  type  of 
agency  they  may  be,  no  diagraiamatic  illustration  is  pre- 
sented  to  demonstrate  the  relationship  between  these  vari¬ 
ables. 

Stages  of  Development 

There  is  also  a  close  rel*iti6nship  between  the  spec¬ 
trum  of  uncertainty  and  the  stages  of  the  developmental 
cycle  insofar  as  the  evaluation  of  R  (  D  activities  is 
concerned.  The  steps  in  the  life-cycle  of  a  project 
usually  commence  with  the  initial  consideration  of  the 
proposed  research,  run  through  the  various  in-process 
research  and  development  activities,  and  conclude  with 
the  completion  or  termination  of  the  project  in  some  sort 
of  research  results,  whether  it  be  a  final  report  of 
research  conducted  or  a  prototype  model  developed  through 
research. 


Various  means  of  evaluation  nay  be  employed  to 
determine  how  effective  are  the  research  efforts  of  the 
organization  throughout  the  developnental  cycle.  Tech¬ 
niques  for  pre-appraising  research  are  primarily  concerned 
with  establishing  a  set  of  values  and  criteria  to  assist 
in  the  evaluation  and  selection  of  alternative  research 
proposals  and  against  which  subsequent  effort  and  results 
may  be  compared  once  a  project  is  inaugurated.  Tech¬ 
niques  for  in-process  measurement  normally  attenpt  to 
analyze  the  current  status  and  degree  of  completion  of 
research  efforts,  comparing  these  against  criteria  initi¬ 
ally  planned  for  the  various  points  in  the  developmental 
cycle.  And,  techniques  for  post-evaluation  of  research 
efforts  try  to  determine  the  effectiveness  of  research 
results  in  comparison  to  pre-established  goals  and  criteria. 

The  methods  of  analysis  range  across  the  qualitative 
to  quantitative  spectrum  of  methodology.  Generally,  how¬ 
ever,  more  quantitative  methods  of  analysis  m*y  be  em¬ 
ployed  for  evaluating  research  after  it  has  been  completed 
or  terminated,  since  more  numerical  data  are  nomally 
available  at  that  point  in  the  dei'elopmental  cycle  whtn 
compared  to  the  earliear  stages  of  research.  At  the 
earlier,  planning  stages  of  the  research,  on  the  other 
hand,  in  pre-appraising  the  potential  of  possible  research 
under  consideration,  relatively  little  numerical  data  are 
available  and  the  more  qualliativo  methods  of  judgi*«At 
must  be  consequently  employed.  The  close  relationship  be¬ 
tween  the  stages  in  the  developmental  cycle,  the  spectrum 
of  uncertainty,  and  the  methods  of  evaluation  is  illus¬ 
trated  in  figure  2. 
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Interrelationships 


Coaparison  of  Fii:ures  I  and  2  readily  reveals  that 
the  level  of  research  endeavor  and  the  staii;e  of  the 
developaental  cycle  are  both  closely  related  to  the  spec- 
triM  of  uncertainty.  Indeed,  these  interrelationships 
•ifht  be  illustrated  as  shown  in  Figure  3.  The  diagram 
is  not  a  graph  in  the  mathematical  sense  of  the  word, 
since  the  levels  of  research  endeavor  and  the  stages  of 
development  are  not  to  be  considered  as  the  abscissa  and 
ordinate  of  a  graphical  relationship.  That  is,  sone 
point  on  the  vertical  axis  showing  a  particular  level 
of  research  effort  should  not  be  viewed  as  correlated 
with  sooe  point  on  the  horizontal  axis  showing  a  given 
stage  in  the  research  life*cycle,  simply  because  respec* 
tive  plots  drawn  from  these  points  to  the  line  showing 
the  spectrum  of  uncertainty  happen  to  intersect  at  that 
coordiMte.  Instead,  the  diagram  is  presented  to  show 
that  there  is  a  relationship  between  some  level  of  re¬ 
search  effort  and  the  degree  of  corresponding  uncertaiiity 
and  that  there  is  also  some  correspondence  between  some 
stage  in  the  developmental  cycle  and  the  corresponding 
degree  of  uncertainty. 

Since  the  more  qualitatively-oriented  nethods  of 
evaluation  will  normally  be  used  at  the  lower  end  of  the 
spectrum  of  uncertainty  to  Judge  the  more  fundamental 
levels  of  research  effort  and  the  earlier  research  steps 
in  the  developmental  cycle,  and  since  the  more  quantita¬ 
tively-oriented  methods  will  usually  be  used  at  the  upper 
or  mere  certain  end  of  the  spectrum  to  analyte  the  devein 
mental  levels  of  reseerch  and  the  later  stages  in  the 
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life'cyclc,  these  relationships  Might  be  better  illustrated 
as  shown  in  Figure  4. 

For  instance,  in  referring  to  the  diagram,  it  is  PiOted 
that  the  developmental  level  of  research  effort  near  the 
close  of  the  life-cycle  of  research  activities  may  be 
analyzed  with  highly  quantitative  techniques,  whereas  the 
purer  level  of  research  in  the  beginning  stage  of  the 
cycle  must  be  evaluated  with  highly  qualitative  methods. 

The  curves  also  reveal,  for  example,  that  mostly  qualita¬ 
tive  methods  must  be  normally  used  to  judge  the  develop¬ 
mental  level  of  research  in  the  earlier  stages  of  the  life- 
cycle  and  to  evaluate  the  pure  level  of  research  even  at 
the  end  of  the  developmental  cycle.  The  diagrammatic  pre¬ 
sentation  employing  the  "indifference"  type  curves  for  the 
methods  of  evaluation  is  particularly  appropriate  in  that 
the  difficulty  in  using  highly  quantitative  techniques  of 
measurement  in  almost  all  situations  is  clearly  illustrated. 

Still  another  way  of  looking  at  the  relationships  be¬ 
tween  the  levels  of  research,  the  developmental  cycle,  and 
the  methods  of  analysis  is  shown  dn  Figure  S,  which  pre¬ 
sents  a  three-dimensional  diagram  of  the  nature  of  these 
interrelationships.  Many  of  the  methods  and  techniques 
employed  in  measuring  and  evaluating  research  endeavor 
and  developmental  activities  were  classifed  within  this 
paper  in  relation  to  their  use  for  pre-appraisal,  in- 
process  measurement,  and  post-evaluation  of  R  ^  D  efforts. 

If  cross-sectional  views  of  the  diagram  in  Figure  S 
were  made  at  selected  intervals  along  the  dimension  re¬ 
lating  to  the  stage  of  development,  at  the  point  of 
pre-appraisal,  at  some  point  of  in-process  measurement, 
and  at  the  point  of  post-evaluation  of  the  research,  the 
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woulti  reveal  the  general  pro{)ortion  of  quantl' 
tative  vis-a-vis  qualitative  aethods  of  evaluation  that 
Might  be  eaployed  for  the  various  levels  of  research 
effort.  Figures  6A,  6E,  and  6C  illustrate  these  cross- 
sectional  views.  For  instance,  in  Figure  6A  relating 
to  the  pre-appraisal  of  research  endeavor,  aostly  qualita¬ 
tive  Methods  of  evaluation  are  eaployed  not  cnly  for 
all  the  pure  level  of  research  but  also  for  a  great  deal 
of  the  developaental  level  of  effort  as  well.  In  post¬ 
evaluation  of  research  endeavor,  shown  in  Figure  6C, 
greater  use  aight  be  aade  of  quantitative  Measures, 
especially  in  evaluating  the  developaental  and  applied 
levels  of  research,  liowever,  as  the  dtagraa  shows,  the 
Majority  of  the  pure  level  of  research  endeavor  would 
still  be  evaluated  by  qualitative  Methods  even  after  the 
research  has  been  coMpleted. 

Thus,  the  diagrsMMatic  presentations  in  Figures  6A, 

I,  and  C,  closely  correspond  to  the  **indif ference  curve" 
type  of  approach  shown  in  Figure  ,4.  All  these  figures 
serve  to  eMphasize  the  relatively  restricted  applicability 
of  the  More  quantitatively-oriented  Methods  of  analysis 
in  evaluating  research  and  scientific  endeavors. 

Concluding  Reuarks 

Generally,  then,  the  following  overall  conclusions 
nay  be  drawn  about  the  Methods  of  evaluating  research 
efforts  and  developaental  activities.  First,  the  quan¬ 
titative  Methods  are  better  used  in  the  applied  and 
developaental  levels  of  research,  but  their  use  in  eval¬ 
uating  pure  and  basic  research  is  of  questionable  value. 
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They  arc  also  normally  of  greater  value  in  the  post¬ 
evaluation  stage  of  research  efforts  than  in  the  pre- 
appraisal  stage  of  the  cycle  where  less  data  are  avail¬ 
able  and  greater  uncertainty  exists.  The  qualitative 
methods  of  evaluation  are  better  suited  for  use  at  the 
fundamental  levels  of  research  and  may  also  be  of  greater 
value  than  the  more  quantitative  techniques  in  the  pre¬ 
appraisal  stages  of  the  developmental  cycle.  They  can 
also  be  used  to  advantage  in  measuring  the  quality  of 
research  results,  in  estimating  the  intangible  products 
of  research  such  as  papers  and  publications,  and  in 
evaluating  the  non-profit  motivated  types  of  research 
such  as  that  conducted  by  government  laboratories  and 
fundamental  research  organizations. 

Restated  once  more,  the  quantitative  vis-a-vis  the 
qualitative  methods  are  at  the  opposite  ends  of  the  meas¬ 
urement  spectrum  and  are  closely  related  to  the  spectrum 
of  uncertainty.  Variations  of  one  or  the  other  methods 
or  combinations  of  the  two  are,  therefore,  generally 
employed,  depending  upon  the  level  of  research  effort 
and  the  stage  of  the  developmental  cycle,  as  well  as  many 
other  situational  factors.  As  one  lending  econometrician, 
Ira  Horowitz,  noted- -after  making  a  comprehensive  review 
and  critique  of  methods  used  for  measuring  the  economic 
impact  of  R  ^  D  efforts--the  degree  of  success  of  a  re¬ 
search  program  is  not  to  be  judged  solely  on  the  extent 
the  program  achieves  or  surpasses  its  own  previously 
established  objectives,  but  rather  in  terms  of  the  multi¬ 
plicity  of  objectives  of  the  firm,  ilorowitz  stated:^®® 

Depending  upon  the  nature  of  the  stated 
objectives,  the  analysis  of  effectiveness  nay 
require  both  quantitative  and  qualitative  judg¬ 
ments,  and  these  judgments  will  necessarily  be 


tempered  by  the  knowledge  that  the  extent  to 
which  the  objectives  have  been  attained  is  a 
function,  not  just  of  the  effectiveness  of  the 
research  program,  but  also  of  the  effective¬ 
ness  of  all  the  other  components  that  make  up 
that  complex  organization,  the  economic  unit. 

One  final  point  requires  especial  reemphasis.  As 
has  been  stated  by  Edward  fi.  Roberts  in  his  book.  The 
Dynamics  of  Research  and  Development .  the  human  element 
is  one  of  the  most  important  variables  in  research  and 
development  systems. 

.  .  •  research  and  development  progress, 
measurement,  and  control  are  all  strongly  de¬ 
pendent  upon  the  human  variable.  Not  only  is 
R  and  D  productivity  subject  to  motivation  in¬ 
fluences.  but  the  subjective  evaluations  that 
rest  at  the  basis  of  any  R  and  D  control  system 
are  also  affected.  Many  problems  in  R  and  1) 
may  come  largely  from  the  McGregor  ’’Theory  X” 
difficulty,  that  is.  the  attempt  to  exert  exter¬ 
nal  control  and  authority  on  research  and  devel¬ 
opment  without  incentives  that  will  motivate 
R  and  D  people  to  strive  toward  the  sane  ends. 

• 

Thus,  it  is  necessary  to  establish  an  optimum  climat 
that  fosters  creativity  and  innovation  with  a  minimum  of 
suspicion  and  distrust  and  a  maximum  of  mutual  respect 
and  integrity  before  evaluation  and  measurement  methods 
may  be  successfully  employed. 

In  conclusion,  because  of  its  basic  nature,  scienti¬ 
fic  productivity  is  extremely  difficult  to  measure.  Rut 
by  using  quantitative  means,  wherever  they  may  be  appro¬ 
priately  employed,  to  assist  in  improving  the  quality  and 
validity  of  subjective  or  qualitative  judgments,  the  effe 
tiveness  of  scientific  performance  may  be  evaluated  so  as 
to  provide  management  with  the  guidance  required  for  the 
direction  and  control  of  research  and  development  efforts 
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Calculated  Riric  Basis  Fomiln  -  used  to  conoute  a  project's 
Masured  against  a  standard  which  states  an  acceptable 


rate  of  return  to  be 
return: 


£.  [■">>] 


1  «  0 


1  =  0 


(0.) 


II  •  THo  project's  rate  of  return  on  total  funds  invested. 

1|  ■  Kat  e8t{ii.ated  incoiac  from  the  prv>Jfcct's  results  in 

any  1th  year. 

B  •  The  last  year  In  which  incomes  arc  expected, 

i  ■  The  year  hence  l.i  which  tho  Incotoo  or  outlAy  will  occur. 

P  ■  Pidwiblllty  of  roc«lvlnf»  the  ith  income, 

0i  •  Ihs  net  increswntal  inveotment  outlay  in  any  ith  year. 


Unadlttctad  Bitum  Tachnioue  •  aakes  broad  calculations  of  a  project's  potential 
rttii.ii  vorris  Its  coats.  Projects  are  sioi'e  dosirablo  as  R  increases,  the 
■IniMia  R  rsqulred  to  accept  any  project  being  30  percent  before  taxes; 

h  <h) 

i  B  0 

ii 

i  •  0 

iditrtt 

R  ■  Ths  rat#  of  return  on  Investiaonts  In  tho  project, 

•  Iho  Incoais  the  project  will  yield  in  each  Ith  year, 

0|  ■  The  increnent*!  outlay  required  in  each  1th  year, 

i  ■  Hit  number  of  ye»ro  hence  each  income  or  outlay  is 

rtallsed, 

n  ■  Ibt  nusdier  of  years  hence  the  last  income  or  outlay 

it  rsflited. 


Horiiold  Try tfpiy tion  ••  coaputta  tht  present  worth  of  the  incomes  tiie  project 
will  yitld  for  tonpariton  with  the  present  value  of  the  project's  actual  and 
projtcitd  ootit  in  order  to  determine  if  the  project  is  financially  accept- 
tbltt 
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p 


where: 


R  f 


0 

R' 

(1  4  R*)” 


P  s  The  present  worth  of  the  incomes  the  project  will 
yield  if  success  Ail. 

0  •  The  average  annual  incremental  income  yielded  if 

the  nroject  is  suecessAil. 

R*  e  The  averaf^  net  return  on  capital  invested  in  the 
enterorise. 

R  e  The  cun^nt  rate  of  Interest  on  investments. 

n  ■  The  nu^nber  of  years  within  which  research  costs 

must  be  recovered. 


Present  Value  Technlmies  -  often  are  used  since  the  (1  •h  R)*^  factor,  shown  below, 
is  readily  available  in  standard  financial  tables  and  projects  can  be  ranked 
in  order  of  ascending  P  to  determine  their  relative  desirability: 

P  =  ^  Fci  (1  t  R)-l1 

1  =  0  ^ 

where: 

P  2  The  project's  present  value. 

■  The  not  cash  inflow  in  each  Hh  year  (net  cash  outflow 
in  a  year  would  make  Ci  negatlvo). 

i  ■  The  number  of  years  hence  the  cash  flow  occurs. 

n  m  The  number  of  y»;aro  hence  the  last  significant  cash 

flow  secure. 

R  e  A  discount  rate  %ihieh  msasiires  the  rate  at  which  the 

ccmiDany  would  invest  its  money  to  obtain  cash  flows 
of  equal  risk  in  the  same  year. 


Lifs^Cvcle  Equation  -  cited  by  Ra?.Tiond  Tillers.  Once  the  pattern  of  the  first 

cycle  of  dovsloisnent,  hMs  been  determined,  the  pattern  for  all  the  subsequent 
cycles  may  be  determined  by  the  followinr  equations: 


y 

■ 

k  (1  -  e"*^^) 

y* 

s 

2  k  a  t 

where: 

y 

a 

Knnoower  to  date.  In  man-months. 

y* 

u 

Manpower  for  each  fnontb. 

k 

m 

Total  effort  for  the  cycle  in  man  months 

•  •  Piar«Bet«r  rovertitR#  ih  pe  ff  ite  curv«, 

t  M  TIm  trxM  •lArt  of  tha  cycle.  If.  xonths, 

•  m  k  eonotont,  the  base  of  n'^turel  lofarithK*  •  2,71d2d... 


JSSlSESi  forailae  •  cited  by  Rayaond  Villers,  IVo  forcuI«e  are  used  to 

•valiMto  th*  Mpectod  coti  in  tiae  «nd  noney  of  pursuing  an  alternative  solu¬ 
tion  to  a  problen.  After  the  ootieun  alternative  is  selected  the  fonsilae 
•OBMrito  the  situation  end  ahow  hou  technical  orogress  la  related  to  actual 
asponditnraa  of  tins  and  aonty.  IHe  fonnila  for  the  tine  factor  ia: 

?t  «  Ot  -  (At  -  E^) 

vliara: 

ft  ®  The  prelieted  tine  variance, 

Ot  ®  Schooled  tine  for  the  project. 

At  *  Tina  already  anent  on  the  project, 

Bt  *  Anount  of  tine  that,  an  the  baala  of  the  revised 

sehedolo,  vlll  be  needed  to  eonplete  the  pro>ct« 

Qy  sdding  ft  to  the  expected  eonoletitn  date,  one  can  show  t;>e  revised  esti  • 
Mtsd  eonplstion  date,  Ihia  revised  estimate  mist  be  ccnpared  to  the  cco- 
oany's  eoNdtnsnts  and  to  the  neoda  of  the  narkct  before  it  can  bo  definitely 
oonsidsrsd  as  soesotabls,  Ihs  second  fonnila  tranalatsa  tha  first  into  dol- 
Isrst 

*b  •  Ob  -  (*b  -  Eb) 

nhsrsi 

f|>  n  ihs  pradieted  variance  freer  the  budget, 

Ob  •  oiiylnal  bid(^t  for  the  project, 

Ab  ■  Ihs  actual  erpenditures  since  the  atart  of  the  project. 

■  Ths  sstinate  cost  of  Et* 

Ihs  varisnes  f^  may  bs  due  to  ths  fact  that  A^  oxceede  the  original  cchodule. 
It  My  also  bs  dus  to  ths  faot  that  ressaroh  paople  have  reported  that  h  ... 
to  bs  rsvlssd.  In  any  oaas,  the  variancs  fb  provides  the  infomation  needed 
and  will  bs  ineludsd  in  ths  rsports  Issued  st  the  various  lavals  of  nanage- 
Mnt, 


OCPARTMCKT  OP  THE  AM  FOACE 
ikm  FOMCC  MkCTiTUTC  OF  TCCMtkOLOCV  «AU» 

W«ttG>«T  rATTCRSOH  AJjR  FOMCC  AASC  OMMO  4»>&y3 
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division  of  the  School  of  SysteM  and  Lofistics,  is  Head  of  the 
Oe|»artaient  of  Research  and  CoiWMnicative  Studies. 

Ho  has  served  as  coaiBander  of  classified  operational  units 
of  the  USAF  Security  Service,  as  a  special  projects  officer 
idchia  Che  Oireccorate  of  Operations,  Headquarters  USAFSS,  as  a 
project  eneioeer  on  the  nation* s  top*prioricy  space  satellite 
systsA  %#ich  the  Office  of  the  Secretary  of  the  Air  Force,  and 
as  the  syscsAS  Aansfienent  officer  for  a  najor  space  vehicle 
proeran  with  Headquarters  Space  Systeoe  Division, 

Lt  Colonel  Quinn  holds  a  aachelor  of  Science  (^ineerinf) 
froA  The  University  of  Tokss,  a  lUister  of  Business  Administration 
•  With  Oiscinccion  (diFineerinB  hanafSAent)  froA  the  Air  Univer* 
sity  Institute  of  Technolopy,  and  a  Doctor  of  Business  AdAinis- 
tratien  CHansgeAont  and  Behavioral  Science)  free  Indiana  Univer* 
sity.  AAons  his  Asny  publications  are  a  series  of  casebooks  in 
lofiistics  AsnsfeAent,  articles  on  pereocinel  sdninlstration,  and 
technical  reports  on  research  and  developpont  ranareront,  econo* 
Aic  concentration  in  industry*  functional  areas  of  euinaferent, 
perforAance  appraisal  and  evaluation,  lof^istics  career  pre^ires* 
Sion*  and  logistics  AansgsAent  education, 

Lt  Colonel  Quinn  holds  the  acadsAic  rank  of  Associate 
Frofessor  of  Logistics  F«anageAent  and  has  received  the  Advanced 
Frofessional  Designation  for  DOD  Logistics  Mnagers  froo  the 
Society  of  Logistics  ^ineers.  He  has  developed  and  taufht 
courses  in  Advanced  F«anagecMnt  and  in  Lopistics  Flannlnr  in  the 
Graduate  Logistics  Kanapement  prograA,  His  honors  and  roAber* 
ships  include  the  Academy  of  FUmagesMnt*  American  Institute  of 
Ftining  and  Ketallurgical  Diu^.inoers*  President’s  Association  of 
the  American  Kanagement  Association*  Arnold  Air  Society,  Beta 
Csssis  SigsA*  Ramshom  atpineering  Society,  Scabbard  and  Blade* 
Sigma  lota  Epsilon*  Society  of  American  Kilitary  Engineers*  and 
Society  of  Logistics  ft^ineers. 


